Synnefo blog has moved

Hello everybody,

the Synnefo blog has moved to www.synnefo.org/blog.

Please update your bookmarks and use our new feed to stay up to date.

Thank you,
the Synnefo team

The second GanetiCon is over and a success

Hello everybody,

as you already know, this September (2/9-4/9), the second annual Ganeti conference took place in Portland, Oregon: GanetiCon 2014. And the Synnefo team was there.

The conference was an overall success, almost the whole Ganeti development team presented their work, all major contributors attended, all the well known users where there, the venue was really nice and Portland itself was beautiful. It was definitely a great choice of city for the conference.

The most interesting thing was finding out for the first time about new users that run Ganeti on big or smaller production environments, meeting them and talking to them in person.

The design discussions went pretty well too, opening up the roadmap for new Ganeti features. There were even cool presents from Google to attendees, such as Chromebook and Nexus devices at the end of the conference!

So, all in all it was an absolutely exciting 3-day event.

For all of you that missed GanetiCon 2014, you can take a glimpse of what happened and also take a look at all presentation slide decks at the GanetiCon site.

After the second year being a success, GanetiCon seems to be here to stay.

Until next year then,
Cheers!

Synnefo @ LinuxCon / CloudOpen North America 2014

Hello everybody,

we are happy to announce that this summer Synnefo will be crossing the Atlantic, landing for the first time officially in the US!

Vangelis Koukis, from the team, will be giving a talk at LinuxCon/CloudOpen North America 2014, introducing Synnefo to the American audience. You can find the complete schedule of the conference here and you can also take a glimpse of the Synnefo presentation here.

So, if you are around, we'd love to see you in Chicago, meet you in person and discuss all things Synnefo, Ganeti and "the Cloud".

On behalf of the Synnefo team,
Constantinos

Synnefo comes to CentOS!

Hello everybody,

As you may already know, Synnefo was initially developed for and deployed on Debian (in the same way Ganeti is). However, since more and more users have been asking to try it out on their own RPM-based distros, we decided to take the time and help them out.

So, today, we are happy to announce that starting Synnefo v0.15.1, alongside Debian packages, we will also be officially providing RPM packages for the whole Synnefo stack. This will be done via our new yum repository:

http://yum.dev.grnet.gr

The first officially supported RPM-based distro, on which we have extensively tested the packages, is CentOS 6.5! For CentOS 6.5, we also provide a new, step by step Installation Guide, describing how to setup the whole Synnefo stack on two nodes. You can find the CentOS guide in the official Synnefo documentation.

Since all the hard work of resolving dependencies, creating packages that existed for Debian but not for Red Hat, and finding out all the little details has been done once for CentOS, we believe that it is now pretty easy for you to have Synnefo running on other RPM-based distros, such as RHEL, Fedora, Scientific Linux, etc., in the near future, too.

As we mention in the CentOS guide, to be able to download and install all Synnefo components you just need to add the GRNET repo on your installation. You can do this by running:

# yum localinstall https://dev.grnet.gr/files/grnet-repo.rpm
# yum update

So, go ahead and try everything out!
As always, any kind of feedback or bug report is highly appreciated.

Enjoy,
on behalf of the Synnefo team,
Constantinos

Synnefo v0.15.1 Released

Hello Synnefo users,

we are happy to announce that Synnefo v0.15.1 is out!

The most notable changes in this version are:

• Support for selectable storage backends for Pithos. Two options available: NFS or Ceph's RADOS.
• Refactoring and extension of the statistics API, to export detailed information about Ganeti clusters, virtual servers, networks, IPv4 pools, images, users, auth providers.
• Introduction of CentOS 6.5 support and new CentOS installation guide.

This version also includes smaller bug fixes and security enhancements. Please see the Changelog file for the complete list.

This is the first Synnefo version that allows running the whole stack over RADOS storage only.

So, feel free to try it out and report feedback.

Enjoy,
on behalf of the Synnefo team,
Constantinos

Synnefo v0.15 Released

Hello Synnefo users,

we are happy to announce that Synnefo v0.15 is out!

This release brings major new features and significant bug fixes across Synnefo.
It focuses mostly on Networking and the changes there affect the whole stack, so you will also see new exciting things on your Web UI.

Note that starting v0.15, Ganeti nodes do not need to have access to the Pithos DB anymore, so this means that Ganeti clusters are now completely separated from the upper Synnefo layer, boosting scalability and security. Additionally, it is now really easy to have Ganeti clusters on geographically distinct locations.

More features are also included, copying/pasting from the NEWS file:

Cyclades

• Networking: Major improvements in the Network Service.  This release introduces a complete implementation of the OpenStack Networking (Neutron) API v2.0, with distinct /networks, /ports, /subnets, and /floatingips API endpoints, replacing the old Cyclades /networks API.
• Networking: Support the creation of arbitrary L2/L3 network topologies, with IPv4-only, dual IPv4/IPv6 or IPv6-only functionality.
• Networking: Simplify network administration, support forced networking configurations, e.g., a permanent management network across VMs.
• Networking: Introduce new CLI management commands for handling networks, subnets, and ports.
• Networking: Support management of floating IPs as a distinct resource in Cyclades. Users may create new IPs from admin-defined IP pools, attach them to and detach them from VMs arbitrarily, and eventually release them.
• Compute: Support server resize, by changing server Flavors. Extend the API and UI accordingly. Support fine-grained, per-flavor control of resize and create actions ("allow_create" flavor attribute).
• Compute: Support finer-grained accounting of "Active CPUs", "Active RAM"  resources, in addition to "CPUs", "RAM". Active resources are those of running VMs. This allows the administrator to limit currently used VM resources to only a subset of the total allocated resources.
• Compute: Support NIC hotplugging. Allow addition and removal of NICs on running VMs, without a reboot, to support on-the-fly network reconfiguration, e.g., allocation of floating IPs, without the need for NAT.
• Compute: Speed up server reconciliation, by performing operations in parallel, across Ganeti backends.
• Compute: Support running snf-vncauthproxy in distinct machine, with control channel over TCP. This simplifies firewalling.
• UI: Numerous UI improvements, especially wrt Networking: Better handling of multiple NICs per server, with multiple IPv4 and IPv6 addresses per NIC. Introduce distinct "IP addresses" view. The "Create New Machine" wizard has also been extended to support on-the-fly networking configuration of a new machine.
• UI: Introduce distinct "SSH key" view.
• UI: Make Google fonts base URL configurable, to support running in networks isolated from the public Internet.
• UI: Support consistent display of pending actions, e.g., "Shutting Down...", or "Destroying...". Previously, they were reset upon page reload.
• UI: Support arbitrary groupings of public networks, based on their name.
• Quota: Support quota enforcement through new 'enforce-resource-cyclades' management command with fine-grained, per-resource actions, to bring current per-user resource usage within specified limits.
• Statistics: Export basic statistics at the /admin/stats/detail API endpoint. Also make them available at the CLI.
• Administration: Support mixed DRBD/Archipelago-based Ganeti backends. Previously, a Ganeti cluster was defined as Archipelago-only explicitly.

 Pithos

• API: Numerous bugfixes and improvements across the codebase. Improve handling of various corner cases, with better error reporting.
• UI: Support separate view/unsafe domain, for increased security. Users may download their files from a specific view domain, distinct from the API domain, to prevent malicious user content from accessing cookies and other sensitive data set at the level of the API domain. This uses newly-introduced support for OAuth2 tokens in Astakos.

Astakos

• Projects: Improved project handling: Better logging of project, application, and membership actions.
• Account API: The Account API has been extended to allow full programmatic handling of projects.
• Authentication: Remove obsolete /account/v1.0/authenticate call in favor of the standard POST /identity/v2.0/tokens Keystone API call.
• Authentication: Support OAuth 2.0 Authorization Code Grant, generate and validate OAuth 2.0 access token. This is used by Pithos to support distinct API and view/unsafe domains.
• Authentication: Support arbitrary attributes as unique identifiers, e.g., eduPersonTargetedId, or ePPN, by extracting them from the ``REMOTE_USER`` header. Also fill third-party signup form fields automatically.
• Administration: Introduce numerous new admin commands at the CLI, simplify resource and user management commands.
• Resources: Finer-grained resource control, with individual setting of resource visibility in the API, the UI, and Projects.
• Statistics: Export basic statistics at the /admin/stats/detail API endpoint. Also make them available at the CLI.

Synnefo-wide

• Burnin tests: Significant improvements to the Synnefo Continuous Integration mechanism (snf-burnin). It has been extended to also test the Pithos storage service.
• Administration: Better error logging, and reporting across Synnefo. Admins are notified by default, via email, whenever an unhandled exception occurs in a Synnefo worker.
• Administration: Ganeti nodes no longer access the Pithos DB at all. Previously, read-only access was required to support spawning VMs from Images on Pithos. As of v0.15, Ganeti nodes only need read-only access to the Pithos block store (NFS or RADOS).

Upgrading to v0.15 is definitely recommended. You can find the corresponding upgrade notes here.

As always feedback is highly appreciated.

Enjoy,
on behalf of the Synnefo team,
Constantinos

Synnefo Architecture

Hello everybody,

On a previous post, almost a year ago, we described the services and REST APIs of Synnefo.

Today, we have 2 new diagrams that show Synnefo's layered approach and overall architecture in more detail. First, we extended the diagram from the previous post that presented the zoom-out view, to include all layers from the hypervisor up to the clients (Web UI or command line). We also updated the API layer to reflect the complete OpenStack compatibility that has been included in the latest version. We have now fully integrated Neutron, Cinder and Keystone along with the previously implemented Nova, Glance and Swift:

The second diagram is created from scratch and zooms in to unveil the whole internal Synnefo architecture, showing all Synnefo components and the interconnections between them:

Note that in the left part (Compute/Network) the cloud layer (Synnefo) interacts with the cluster layer (Ganeti clusters) only via HTTP in the effect path and only via AMQP in the update path. The cluster layer does not access any DB, allowing us to scale linearly by adding new nodes on a Ganeti cluster or new Ganeti clusters altogether, even in geographically distinct locations.

Also note, that with Archipelago one can operate over multiple storage technologies
transparently, without the need to change anything on the upper layers, may these be Ganeti or Pithos.

Just to give you a preview on what's coming next, along with many new features, starting v0.16, Pithos will become a thin layer on top of Archipelago or in other words Archipelago will become the backend for Pithos. This will allow for 2 things:

• Complete unification of cloud storage as seen by the upper layers
• Complete abstraction of the actual data store underneath

So, hope you find the above interesting and hope the diagrams help everybody to understand the Synnefo architecture a little better.

Thanks,
Constantinos