DidRocks' blog

apt-get install freedom

Aller au contenu | Aller au menu | Aller à la recherche

mardi, septembre 1 2015

Ubuntu Make 15.09 featuring experimental Unity 3D editor support

Last thurday, the Unity 3D team announced providing some experimental build of Unity editor to Linux.

This was quite an exciting news, especially for me as a personal Unity 3D user. Perfect opportunity to implements this install support in Ubuntu Make, and this is now available for download! The "experimental" comes from the fact that it's experimental upstream as well, there is only one version out (and so, no download section when we'll always fetch latest) and no checksum support. We talked about it on upstream's IRC channel and will work with them on this in the future.

Unity3D editor on Ubuntu!

Of course, all things is, as usual, backed up with tests to ensure we spot any issue.

Speaking of tests, this release as well fix Arduino download support which broke due to upstream versioning scheme changes. This is where our heavy tests investment really shines as we could spot it before getting any bug reports on this!

Various more technical "under the wood" changes went in as well, to make contributors' life easier. We got recently even more excellent contributions (it's starting to be hard for me to keep up with them to be honest due to the load!), more on that next week with nice incoming goodies which are cooking up.

The whole release details are available here. As usual, you can get this latest version direcly through its ppa for the 14.04 LTS, 15.05 and wily ubuntu releases.

Our issue tracker is full of ideas and opportunities, and pull requests remain opened for any issues or suggestions! If you want to be the next featured contributor and want to give an hand, you can refer to this post with useful links!

jeudi, août 13 2015

Ubuntu Make 15.08 with Scala support and a visual studio code hot fix community story

Here is a little bit of the start of my day:

As usual, I open the Ubuntu Make large test suite running continuously against trunk and latest release package. I saw that since yesterday 7PM CEST Visual Studio Code page changed its syntax and is not downloadable anymore by Ubuntu Make.

Jumping on the github's project page, I saw a couple of bugs opened about it, and as well a pull request to fix this from a new contributor, Vartan Simonian! All this in less than 12 hours of this breakage. I just had to merge it, changing the medium tests and cut a release. Hey community work!

That made my day, it was thus high time to release this new Ubuntu Make 15.08. Notice that we are starting to follow the scheme "YY.MM" which is quite handy for versioning this kind of continously evolving projects.

Scala logo In addition to this fix, you will notice that Igor Vuk added scala support. Your always fresh-willingness of scala will now be satisfied through Ubuntu Make!

Some other fixes (progress bar out of range by Anton Antonov, new pep8 release issues found) are also part to make this a great release… And we have even more in the pipe! Thanks again to all our Ubuntu Make contributors, this makes working on this project an awesome journey!

As usual, you can get this latest version direcly in Ubuntu Wily, and through its ppa for the 14.04 LTS, 15.05 ubuntu releases.

Our issue tracker is full of ideas and opportunities, and pull requests remain opened for any issues or suggestions! If you want to be the next featured contributor and want to give an hand, you can refer to this post with useful links!

mardi, août 4 2015

Ubuntu Make 0.9.2 hot from the builders, with Firefox Developer Edition language support

Ubuntu Make 0.9.2 has just been released and features language support in our Firefox Developer Edition installation!

Thanks to our new awesome community contributor Omer Sheikh, Ubuntu Make now enables developers to install Firefox Developer Edition in their language of choice! This is all backed with our mandatory medium and large extensive testsuites. Big thanks to him for getting that through!

The installation process will ask you (listing all available languages) what is your preference for that framework:

 $ umake web firefox-dev
 Choose installation path: /home/didrocks/tools/web/firefox-dev
 Choose language: (default: en-US)
 ach/af/sq/ar/an/hy-AM/as/ast/az/eu/... fr
 Downloading and installing requirements
 100% |#########################################################################|
 Installing Firefox Dev
 |##############################################################################|
 Installation done

And here we go, with Firefox Dev Edition installed in french:

Firefox Developer Edition en français svp!

You can as well use the new --lang= option to do that in non interactive mode, like scripts.

Brian P. Sizemore joined as well the Ubuntu Make contributor crew with this release with some clarification of our readme page. Valuable contribution to all newcomers, thanks to him as well!

Some general fixes as well were delivered into this new release, full list is available in the changelog.

As usual, you can get this latest version direcly in Ubuntu Wily, and through its ppa for the 14.04 LTS, 15.05 ubuntu releases.

Our issue tracker is full of ideas and opportunities, and pull requests remain opened for any issues or suggestions! If you want to be the next featured contributor and want to give an hand, you can refer to this post with useful links!

mardi, juillet 21 2015

Arduino support and various fixes in Ubuntu Make 0.9

A warm summer has started in some part of the world and holidays: beach and enjoying various refreshements!

However, the unstoppable Ubuntu Make team wasn't on a pause and we continued making improvements thanks to the vibrant community around it!

What's new in this release? First Arduino support has been added with the vast majority of work done by Tin Tvrtković. Thanks to him for this excellent work! To be able to install arduino IDE, just run:

$ umake ide arduino

Note that your user will eventually be added to the right unix group if it was not already in. In that case, it will ask you to login back to be able to communicate with your arduino device. Then, you can enjoy the arduino IDE:

Arduino IDE

Some other hilights from this release is the deprecation of the Dart Editor framework and replacement by Dart SDK one. As of Dartlang 1.11, the Dart Editor isn't supported and bundled anymore (it still exists as an independent eclipse plugin though). We thus marked the Dart Editor framework for removal only and added this Dart SDK (adding the SDK to the user's PATH) instead. This is the new default for the Dart category.

Thanks to our extensive tests, we saw that the 32 bits of Visual Studio Code page changed and wasn't thus installable anymore. It's as fixed as of this release.

A lot of other improvements (in particular in the tests writing infra and other minor fixes) are also part of 0.9. A more detailed changelog is available here.

0.9 is already available in Wily, and through its ppa, for 14.04 LTS and 15.04 ubuntu releases! Get it while it's warmhot!

jeudi, avril 30 2015

Ubuntu Make 0.7 released with Visual Studio Code support

If you followed recent news, yesterday Microsoft announced Visual Studio Code support on stage during their Build conference. One of the nice surprise was that this new IDE, focused on web and cloud platforms, is available on Mac OS X and of course, on Linux! Some screenshots were presented at the conference with Visual Studio Code running on Ubuntu in an Unity Session.

This sounded like a nice opportunity for Ubuntu Make to shine again, and we just added this new support! And yeah, it's a snappy feeling to get it delivered as fast! This release of course brings as well the required non regression large and medium tests to ensure we can track that the installation is working as expected as time pass by and detect any server-side or client-side regression.

To install it, just run:

$ umake web visual-studio-code

Here is the required screenshot of a fresh Visual Studio Code installation with Ubuntu Make!

Visual Studio Code

You can get Ubuntu Make 0.7 through its ppa, for the 14.04 LTS, 14.10 and 15.05 ubuntu releases.

Our issue tracker is full of ideas and opportunities, and pull requests remain opened for any issues or suggestions! For all the various form of contributions and how to give an hand, you can refer to this post!

mercredi, février 18 2015

Ubuntu Make community releases 0.6 with 5 new supported platforms

What always inspires me in my work is that community participation is at heart of what we are doing, and that’s what help us waking up everyday. Anybody can dive and fix small typos, bugs, or even bring big features to the table! This is exactly what happened with this new Ubuntu Make 0.6 release, entirely backed by community participation!

In addition to zsh support, you will find 5 new jetbrains IDE supported into the “ide” Ubuntu Make category, which are: RubyMine, PyCharm educational edition, PyCharm professional, WebStorm and PhpStorm! Those new ones align nicely and complete the already supported 10 platforms in Ubuntu Make, totalising thus (if I can do maths) 15 of them! Remember we started the vivid cycle with only one platform supported as it was about building solid foundations and helpers in the tool so that such contributions can exist. :) I guess there has never been a better time to be a developer using Ubuntu as their OS!

I was delighted to see such big contributions being posted as a pull requests by Anton Antonov without much more to do afterwards than slight adjustments. This testified that the overall frameworks (and tests infra as well) is easy enough to grock and that’s a huge source of satisfaction for us! I was even more excited to see that another community member who did a lot of work on Ubuntu Make (Tin Tvrtković) helped on the review and gave great advice!

Great work team! That’s the kind of things that made my day. All those niceties and excellent contributions are available as of now in Ubuntu Make 0.6 in Vivid, as well, through its ppa, to 14.04 LTS and 14.10 ubuntu releases.

Maybe you can be the next awesome contributor? Our issue tracker is full of ideas and opportunities, and pull requests remains opened for any issues or suggestions! For all the various form of contributions and how to give an hand, you can refer to this post!

jeudi, février 12 2015

Ubuntu Make 0.5 adds four new platforms

Hot from the builders, we just cut a big release of Ubuntu Make 0.5 bringing fresh new support to a bunch of IDEs and programming languages! Web developers will see great enhancements in this release, but other developers are not left out!

We welcome first a new “web” category which is hosting Dart support now! Dartlang fans would be able to get the editor and sdk installed, registered in the OS and added to the launcher in a simple command!

umake dart

and you are ready to go!

Firefox Developer Edition is now available as well under the same new “Web” category. Installing and getting a great environment for your web development is just a command away on Ubuntu!

The JetBrains suite got its Idea Ultimate edition support thanks to another excellent contribution from Tin Tvrtković. He already fixed other jetbrains support if you were redirected (based on location) over the ftp download.

Another request was to add Android NDK support. Your wish is now fulfilled under the Android category!

In addition to this, we merged latest translations coming from the community, thanks to all of you contributing to it! Please note that due to those new categories, there are new strings to be translated. It’s a nice way to get your first contribution to ubuntu make just awaiting for you!

Ubuntu Make 0.5 is already available in Vivid, as well, through its ppa, to 14.04 LTS and 14.10 ubuntu releases. For full details about the release are available here.

Our issue tracker and pull requests remains opened for any issues or suggestions!

jeudi, janvier 22 2015

Bringing appmenu support for java application and Ubuntu Make 0.4.1 with an Intellij IDEA fix

Today we released Ubuntu Make 0.4.1 which validates the application menu support for some java application using swing (like Intellij, Android Studio…) and fixes Intellij IDEA support.

Vertical screen estate is particularly valuable for developers, to maximize the place where you can visualize your code and not bother too much about the shell itself. Also, in complex menu structure, it can be hard to find the relevant items. Unity introduced a while back (2010!) the excellent application menu and then grows the HUD support to search through those menus. We even got recently new options for menu integration without renouncing on those principles. However, until now, some java-based IDEs didn't get default appmenu and HUD support. It was time to get that fixed with our Ubuntu Loves Developers focus!

Appmenu support in intellij IDEA

The application menu support is installed by default on Ubuntu Vivid thanks to our work with jayatana's excellent contributor Dan Jared! We did some cleaning and worked with him to get jayatana into ubuntu vivid, and then, promote it on the Ubuntu Desktop image[1]. On older releases, we pushed jayatana into the Ubuntu Make ppa and every new install through that tool will install as well this support as needed.

We also saw jetbrains changing their download page structure, making Intellij IDEA not being installable anymore. Less than 24 hours after a bug report being opened, we got this new 0.4.1 release including a fix from Intellij IDEA support contributor to Ubuntu Make, Tin Tvrtković. Big kudos to him for the prompt reaction! The tests have also been updated to reflect this new page structure.

Those nice goodies and fixes are available on ubuntu vivid (ensure you install Ubuntu Make 0.4.1), and as well, through its ppa for 14.04 LTS and 14.10 ubuntu releases. Keep in mind that you need to restart your session once jayatana has been installed to get the menu export to Unity available.

Another release demonstrating how the Ubuntu and Ubuntu Make community really work well together, get things quickly under control and shine! If you want to help out defining and contributing in making Ubuntu the best platform for developers, please head here!

Note

[1] which won't install java on the image by default, don't be scared ;)

mardi, janvier 6 2015

Ubuntu Make 0.4 starts the new year adding Go support

Ubuntu Make 0.4 has just been released and brings Go support and a new game category!

To hack using Go under Ubuntu, just open a terminal and type:

umake go

and here we "go"! This will enable developers to always install the latest Google golang version and setting up some needed environment variables for you.

We also starts thinking about game developers. Putting the code where our mouth is, we are pleased to inaugurate a new "games" section, bringing stencyl, an amazing quick and easy way to make games for multiple platforms available!

umake games stencyl

and you will be able to be creative in creating the new top seller game!

Ubuntu Make 0.4 is already available in Vivid, as well, through its ppa, to 14.04 LTS and 14.10 ubuntu releases.

If you have any idea (like a favorite IDE for go!) or some other game platforms, our issue tracker is opened for any suggestion!

On other news, the new name migration is now over with the github repository being moved under the ubuntu namespace and is now available here, waiting eagerly for your contribution!

lundi, novembre 24 2014

Ubuntu Developer Tools needs you for its new name!

We’ve been talking about the Ubuntu Developer Tools Center for a few months now. We’ve seen a lot of people testing it out & contributing and we had a good session at the Ubuntu Online Summit about what the near future holds for UDTC.

Also during that session, emerging from feedback we received we talked about how “UDTC” and “Ubuntu Developer Tools Centre” is a bit of mouthfull, and the acronym is quite easy to muddle. We agreed that we needed a new name, and that’s where we need your help.

We’re looking for a name which succinctly describes what the Developer Tools Center is all about, its values and philosophy. Specifically, that we are about developing ON Ubuntu, not just FOR Ubuntu. That we strive to ensure that the tools made available via the tools center are always in line with latest version delivered by the upstream developers. That we automate the testing and validation of this, so developers can rely on us. And that use LTS releases as our environment of choice so developers have a solid foundation on which to build. In a nutshell, a name that conveys that we love developers!

If you have a great idea for a new name please let us know by commenting on the Google+ post or by commenting on this blog post.

The final winner will be chosen by a group of Ubuntu contributors but please +1 your favorite to help us come up with a shortlist. The winner will receive the great honor of an Ubuntu T Shirt and knowing that they have changed history! We’ll close this contest by Monday 8th of December.

Now, it’s all up to you! If you want to also contribute to other parts of this ubuntu loves developers effort, you’re more than welcome!

mardi, novembre 4 2014

Just released Ubuntu Developer Tools Center 0.1.1

In a nutshell, this release is fixing the changes introduced by the new Android Studio (0.8.14) download in beta channel.

The web page changed from a md5 checksum to a sha1. We do that check as we grab that information from a secure https connexion, but then, we download from dl.google.com which isn't https enabled.

Thanks again to Tin Tvrtković who did the work to add this support and makes the necessary changes! :) I just had to do a little cleanup and then releasing it!

How did we noticed it?

Multiple source of inputs:

  • The automated tests which are running every couple of hours were both failing reliably in trunk and latest release version, on both architecture, only for android studio in the large tests (medium tests, with fake assets were fine). This clearly meant that a third-party broke us!
  • Some bug reports from the community
  • Some directly addressed emails!

It was a nice way to see that Ubuntu Developer Tools Center is quite used, even if this is through an external breakage. Thanks to all those inputs, the fix went in shortly!

On Android Studio not shipping the sdk anymore.

With this new release, Android Studio is not shipping the sdk embedded anymore and it needs to be downloaded separately. I think this is a nice move: decoupling editor code from sdk tools just makes sense.

However, in my opinion, the initial setup seems a little bit not obvious for people who just want to start android development (you have to download the sdk tools, and then, find in Android Studio where to set that path to the sdk, knowing that reopening the same window will keep you previous path only you have one sdk version installed from those sdk tools).

It seems as well that there is no way to set that up from the command line or configuration file without listing installed sdk versions, which will then not be a robust way for the ubuntu developer tools center to do it for you.

So, for now, we are following the upstream (I think they have a plan for better integration in the future) way of letting the user downloading the sdk, I will be interested in getting feedback from new users on how it went for them on that step.

Availability

0.1.1 is now available to the regular channels: - in vivid - in the ubuntu-developer-tools-center ppa for Ubuntu 14.04 LTS and 14.10.

mercredi, octobre 29 2014

Eclipse and android adt support now in Ubuntu Developer Tools Center

Eclipse and Android ADT support now in Ubuntu Developer Tools Center

Now that the excellent Ubuntu 14.10 is released, it's time to focus as part of our Ubuntu Loves Developers effort on the Ubuntu Developer Tools Center and cutting a new release, bringing numerous new exciting features and framework support!

0.1 Release main features

Eclipse support

Eclipse is now part of the Ubuntu Developer Tools Center thanks to the excellent work of Tin Tvrtković who implemented the needed bits to bring that up to our users! He worked on the test bed as well to ensure we'll never break unnoticed! That way, we'll always deliver the latest and best Eclipse story on ubuntu.

To install it, just run:

$ udtc ide eclipse

and let the system set it up for you!

eclipse.png

Android Developer Tools support (with eclipse)

The first release introduced the Android Studio (beta) support, which is the default in UDTC for the Android category. In addition to that, we now complete the support in bringing ADT Eclipse support with this release.

eclipse-adt.png

It can be simply installed with:

$ udtc android eclipse-adt

Accept the SDK license like in the android studio case and be done! Note that from now on as suggested by a contributor, with both Android Studio and Eclipse ADT, we add the android tools like adb, fastboot, ddms to the user PATH.

Ubuntu is now a truly first-class citizen for Android application developers as their platform of choice!

Removing installed platform

As per a feature request on the ubuntu developer tools center issue tracker, it's now really easy to remove any installed platform. Just enter the same command than for installing, and append --remove. For instance:

$ udtc android eclipse-adt --remove
Removing Eclipse ADT
Suppression done

Enabling local frameworks

As requested as well on the issue tracker, users can now provide their own local frameworks, by using either UDTC_FRAMEWORKS=/path/to/directory and dropping those frameworks here, or in ~/.udtc/frameworks/.

On glorious details, duplicated categories and frameworks loading order is the following:

  1. UDTC_FRAMEWORKS content
  2. ~/.udtc/frameworks/ content
  3. System ones.

Note though that duplicate filenames aren't encouraged, but supported. This will help as well testing for large tests with a basic framework for all the install, reinstall, remove and other cases common in all BaseInstaller frameworks.

Other enhancements from the community

A lot of typo fixes have been included into that release thanks to the excellent and regular work of Igor Vuk, providing regular fixes! A big up to him :) I want to highlight as well the great contributions that we got in term of translations support. Thanks to everyone who helped providing or updating de, en_AU, en_CA, en_GB, es, eu, fr, hr, it, pl, ru, te, zh_CN, zh_HK support in udtc! We are eager to see what next language will enter into this list. Remember that the guide on how to contribute to Ubuntu Developer Tools Center is available here.

Exciting! How can I get it?

The 0.1 release is now tagged and all tests are passing (this release brings 70 new tests). It's available directly on vivid.

For 14.04 LTS and 14.10, use the ubuntu-developer-tools-center ppa where it's already available.

Contributions

As you have seen above, we really listen to our community and implement & debate anything coming through. We start as well to see great contributions that we accept and merge in. We are just waiting for yours!

If you want to discuss some ideas or want to give a hand, please refer to this blog post which explains how to contribute and help influencing our Ubuntu loves developers story! You can as well reach us on IRC on #ubuntu-desktop on freenode. We'll likely have an opened hangout soon during the upcoming Ubuntu Online Summit as well. More news in the coming days here. :)

mercredi, septembre 24 2014

Ubuntu Developer Tools Center: how do we run tests?

We are starting to see multiple awesome code contributions and suggestions on our Ubuntu Loves Developers effort and we are eagerly waiting on yours! As a consequence, the spectrum of supported tools is going to expand quickly and we need to ensure that all those different targeted developers are well supported, on multiple releases, always delivering the latest version of those environments, at anytime.

A huge task that we can only support thanks to a large suite of tests! Here are some details on what we currently have in place to achieve and ensure this level of quality.

Different kinds of tests

pep8 test

The pep8 test is there to ensure code quality and consistency checking. Tests results are trivial to interpret.

This test is running on every commit to master, on each release during package build as well as every couple of hours on jenkins.

small tests

Those are basically unit tests. They are enabling us to quickly see if we've broken anything with a change, or if the distribution itself broke us. We try to cover in particular multiple corner cases that are easy to test that way.

They are running on every commit to master, on each release during package build, every time a dependency is changed in Ubuntu thanks to autopkgtests and every couple of hours on jenkins.

large tests

Large tests are real user-based testing. We execute udtc and type in stdin various scenarios (like installing, reinstalling, removing, installing with a different path, aborting, ensuring the IDE can start…) and check that the resulting behavior is the one we are expecting.

Those tests enables us to know if something in the distribution broke us, or if a website changed its layout, the download links are modified, or if a newer version of a framework can't be launched on a particular Ubuntu version or configuration. That way, we are aware, ideally most of the time even before the user, that something is broken and can act on it.

Those tests are running every couple of hours on jenkins, using real virtual machines running an Ubuntu Desktop install.

medium tests

Finally, the medium tests are inheriting from the large tests. Thus, they are running exactly the same suite of tests, but in a Docker containerized environment, with mock and small assets, not relying on the network or any archives. This means that we ship and emulate a webserver delivering web pages to the container, pretending we are, for instance, https://developer.android.com. We then deliver fake requirements packages and mock tarballs to udtc, and running those.

Implementing a medium tests is generally really easy, for instance:

class BasicCLIInContainer(ContainerTests, test_basics_cli.BasicCLI):

"""This will test the basic cli command class inside a container"""

is enough. That means "takes all the BasicCLI large tests, and run them inside a container". All the hard work, wrapping, sshing and tests are done for you. Just simply implement your large tests and they will be able to run inside the container with this inheritance!

We added as well more complex use cases, like emulating a corrupted downloading, with a md5 checksum mismatch. We generate this controlled environment and share it using trusted containers from Docker Hub that we generate from the Ubuntu Developer Tools Center DockerFile.

Those tests are running as well every couple of hours on jenkins.

By comparing medium and large tests, as the first is in a completely controlled environment, we can decipher if we or the distribution broke us, or if a change from a third-party changing their website or requesting newer version requirements impacted us (as the failure will only occurs on the large tests and not in the medium for instance).

Running all tests, continuously!

As some of the tests can show the impact of external parts, being the distribution, or even, websites (as we parse some download links), we need to run all those tests regularly[1]. Note as well that we can experience different results on various configurations. That's why we are running all those tests every couple of hours, once using the system installed tests, and then, with the tip of master. Those are running on various virtual machines (like here, 14.04 LTS on i386 and amd64).

By comparing all this data, we know if a new commit introduced regressions, if a third-party broke and we need to fix or adapt to it. Each testsuites has a bunch of artifacts attached to be able to inspect the dependencies installed, the exact version of UDTC tested here, and ensure we don't corner ourself with subtleties like "it works in trunk, but is broken once installed".

jenkins test results

You can see on that graph that trunk has more tests (and features… just wait for some days before we tell more about them ;)) than latest released version.

As metrics are key, we collect code coverage and line metrics on each configuration to ensure we are not regressing in our target of keeping high coverage. That tracks as well various stats like number of lines of code.

Conclusion

Thanks to all this, we'll probably know even before any of you if anything is suddenly broken and put actions in place to quickly deliver a fix. With each new kind of breakage we plan to back it up with a new suite of tests to ensure we never see the same regression again.

As you can see, we are pretty hardcore on tests and believe it's the only way to keep quality and a sustainable system. With all that in place, as a developer, you should just have to enjoy your productive environment and don't have to bother of the operation system itself. We have you covered!

Ubuntu Loves Developers

As always, you can reach me on G+, #ubuntu-desktop (didrocks) on IRC (freenode), or sending any issue or even pull requests against the Ubuntu Developer Tools Center project!

Note

[1] if tests are not running regularly, you can consider them broken anyway

mercredi, septembre 10 2014

How to help on Ubuntu Developer Tools Center

Last week, we announced our "Ubuntu Loves Developers" effort! We got some great feedback and coverage. Multiple questions arose around how to help and be part of this effort. Here is the post to answer about this :)

Our philosophy

First, let's define the core principles around the Ubuntu Developer Tools Center and what we are trying to achieve with this:

  1. UDTC will always download, tests and support the latest available upstream developer stack. No version stuck in stone for 5 years, we get the latest and the best release that upstream delivers to all of us. We are conscious that being able to develop on a freshly updated environment is one of the core values of the developer audience and that's why we want to deliver that experience.
  2. We know that developers want stability overall and not have to upgrade or spend time maintaining their machine every 6 months. We agree they shouldn't have to and the platform should "get out of my way, I've got work to do". That's the reason why we focus heavily on the latest LTS release of Ubuntu. All tools will always be backported and supported on the latest Long Term Support release. Tests are running multiple times a day on this platform. In addition to this, we support, of course, the latest available Ubuntu Release for developers who likes to live on the edge!
  3. We want to ensure that the supported developer environment is always functional. Indeed, by always downloading latest version from upstream, the software stack can change its requirements, requiring newer or extra libraries and thus break. That's why we are running a whole suite of functional tests multiple times a day, on both version that you can find in distro and latest trunk. That way we know if:
  • we broke ourself in trunk and needs to fix it before releasing.
  • the platform broke one of the developer stack and we can promptly fix it.
  • a third-party application or a website changed and broke the integration. We can then fix this really early on.

All those tests running will ensure the best experience we can deliver, while fetching always latest released version from upstream, and all this, on a very stable platform!

Sounds cool, how can I help?

Reports bugs and propose enhancements

The more direct way of reporting a bug or giving any suggestions is through the upstream bug tracker. Of course, you can always reach us out as well on social networks like g+, through the comments section of this blog, or on IRC: #ubuntu-desktop, on freenode. We are also starting to look at the #ubuntulovesdevs hashtag.

The tool is really to help developers, so do not hesitate to help us directing the Ubuntu Developer Tools Center on the way which is the best for you. :)

Help translating

We already had some good translations contributions through launchpad! Thanks to all our translators, we got Basque, Chinese (Hong Kong), Chinese (Simplified), French, Italian and Spanish! There are only few strings up for translations in udtc and it should take less than half an hour in total to add a new one. It's a very good and useful way to contribute for people speaking other languages than English! We do look at them and merge them in the mainline automatically.

Contribute on the code itself

Some people started to offer code contribution and that's a very good and motivating news. Do not hesitate to fork us on the upstream github repo. We'll ensure we keep up to date on all code contributions and pull requests. If you have any questions or for better coordination, open a bug to start the discussion around your awesome idea. We'll try to be around and guide you on how to add any framework support! You will not be alone!

Write some documentation

We have some basic user documentation. If you feel there are any gaps or any missing news, feel free to edit the wiki page! You can as well merge some of the documentation of the README.md file or propose some enhancements to it!

To give an easy start to any developers who wants to hack on udtc iitself, we try to keep the README.md file readable and up to the current code content. However, this one can deviate a little bit, if you think that any part missing/explanation requires, you can propose any modifications to it to help future hackers having an easier start. :)

Spread the word!

Finally, spreading the word that Ubuntu Loves Developers and we mean it! Talk about it on social network, tagging with #ubuntulovesdevs or in blog posts, or just chatting to your local community! We deeply care about our developer audience on the Ubuntu Desktop and Server and we want this to be known!

uld.png

For more information and hopefully goodness, we'll have an ubuntu on air session session soon! We'll keep you posted on this blog when we have final dates details.

If you felt that I forgot to mention anything, do not hesitate to signal it as well, this is another form of very welcome contributions! ;)

I'll discuss next week how we maintain and runs tests to ensure your developer tools are always working and supported!

mardi, septembre 2 2014

Ubuntu loves Developers

Ubuntu is one of the best Linux platforms with an awesome desktop for regular users (and soon phone and tablets and more!) and great servers for system administrators and devops. A number of developers are choosing Ubuntu as their primary development system of choice, even if they develop for platforms other than Ubuntu itself, like doing some Android development, web development and so on.

uld.png

However, even if we fill the basic needs for this audience, we decided a few months ago to start a development and integration effort to make those users completely feel at home. Ubuntu loves developers and we are going to showcase it by making Ubuntu the best available developer platform!

Sounds great! What's up then?

We decided to start by concentrating on Android developers. We'll ramp up afterwards on other use cases like Go developers, web developers, Dart… but we want to ensure we deliver a stunning experience for each targeted audience before moving on to the next topic.

After analyzing how to setup an Android development machine on Ubuntu we realized that, depending on the system, it can takes up to 9 different steps to get proper IDE integration and all the dependencies installed. The whole goal was to reduce that to one single command!

Concretely speaking, we created the Ubuntu Developer Tools Center, a command line tool which allows you to download the latest version of Android Studio (beta), alongside the latest Android SDK, and all the required dependencies (which will only ask for sudo access if you don't have all the required dependencies installed already), enable multi-arch on your system if you are on a 64 bit machine, integrate it with the Unity launcher…

launcher-integration.png

As said, we focused on Android Studio (based itself on Intellij IDEA) for now as it seems that’s where Google has been focusing its Android tools development effort for over a year. However, the system is not restrictive and it will be relatively trivial in the near future to add ADT support (Android Development Tools using Eclipse)[1].

android-studio.png

Indeed, The Ubuntu Developer Tools Center is targeted as being a real platform for all developer users on Ubuntu. We carefully implemented the base platform with a strong technical foundation, so that it's easily extensible and some features like the advanced bash shell completion will even make more sense once we added other development tools support.

Availability

We will always target first the latest Ubuntu LTS version alongside the latest version in development. Yes! It means that people who want to benefit for the extensively tested and strong base experience that a Long Term Support version offers will always be up to date on their favorite developer tools and be first-class citizen. We strongly believe that developers always want the latest available tools on a strong and solid stable base and this is one of the core principle we are focusing on.

For now, the LTS support is through our official Ubuntu Developer Tools Center ppa, but we plan to move that to the backports archive with all the newly or updated libraries. For Utopic, it's already available in the 14.10 Ubuntu archive.

Initial available version

Be aware that the Ubuntu Developer Tools Center is currently in alpha. This tool will evolve depending on your feedback, so it's up to you to suggest the direction you want it to go! A blog post on how to contribute will follow in the next days. This initial version is available in English, French and Chinese!

Another blog post will expand as well how we test this tool. For now, just be aware that the extensive test suite is running daily, and it ensures that on all supported platforms we don't break, that the Ubuntu platform itself doesn't break us, or that any 3rd party on which we rely on (like website links and so on) don't change without us spotting it. This will ensure that our tools is always working, with limited downtime.

Example: how to install Ubuntu Developer tools and then, Android Studio

Ubuntu Developer Tools Center

If you are on Ubuntu 14.04 LTS, first, add the UDTC ppa:

$ sudo add-apt-repository ppa:didrocks/ubuntu-developer-tools-center $ sudo apt-get update

Then, installing UDTC:

$ sudo apt-get install ubuntu-developer-tools-center

How to install android-studio

Simply executes[2]:

$ udtc android

And then, accept the installation path and Google license. It will download, install all requirements alongside Android Studio and latest android SDK itself, then configure and fit it into the system like by adding an Unity launcher icon…

And that's it! Happy Android application hacking on Ubuntu. You will find the familiar experience with the android emulator and sdk manager + auto-updater to always be on the latest. ;)

android-sdk-tools.png

Feedback

We welcome any ideas and feedback, as well as contributions as we'll discuss more in the next post. Meanwhile, do not hesitate to reach me on IRC (didrocks on freenode, #ubuntu-desktop as the primary channel to discuss), or on Google+. You can as well open bugs on the launchpad project or github one

I'm excited about this opportunity to work on the developer desktop. Ubuntu loves Developers, and it's all on us to create a strong developer community so that we can really make Ubuntu, the developer-friendly platform of choice!

Notes

[1] For the technical audience, the Android Studio specific part is less than 60 lines

[2] android-studio is the default for the android development platform, you can choose it explicitely by executing "$ udtc android android-studio". Refer to --help or use the bash completion for more help and hints

jeudi, novembre 21 2013

Ubuntu Developer Summit: résumé et discussion avec la communauté francophone. En direct, ce soir, 21h (heure française)

Bonjour à tous,

Suite à une discussion lors de la dernière Ubuntu Party à Paris, une proposition a été faite de résumer ce qui a été débattu pendant l'Ubuntu Developer Summit en ligne qui a eu lieu de mardi à jeudi soir.

Pour rappelle l'UDS est le lieu (maintenant complètement en ligne) où toutes les décisions techniques concernant le développement d'Ubuntu sont prises. Tout le monde est invité à participer par conférence vidéo ou directement sur IRC. Cependant, la barrière de la langue est le frein principal pour nombre de francophones.

Nous allons donc tâcher de résumer ce qui a été couvert par ce vUDS, ainsi que de discuter d'autres sujets, comme l'iso francophone d'ubuntu incluant le Dash Privacy Interface. Nous retrouverons lors de cet événement vidéo en ligne:

  • Frédéric Mandé (quesh, membre du Conseil d'administration de l'association Ubuntu-fr)
  • Loïc Minier (lool, technical architect d'Ubuntu chez Canonical)
  • Rick Spencer (rickspencer3, Vice-président d'Ubuntu Engineering chez Canonical)
  • Rudy ANDRÉ (cm-t, membre du Conseil d'administration de l'association Ubuntu-fr et développeur du Dash Privacy Interface)
  • et moi-même, Didier Roche (didrocks, avec la casquette Ubuntu-fr/Canonical)

Nous utiliserons la même technologie que pour le vUDS, c'est à dire des google hangout. Comment nous suivre?

C'est (presque) très simple. Ceux qui me suivent sur Google+ doivent avoir vu cet évènement. Une vidéo (Youtube) apparaitra quelques minutes avant le début de la diffusion. Je copierai à ce moment là également le lien sur le topic des nouveautés de trusty dans le forum ubuntu-fr.

Si vous souhaitez poser des questions/participer, vous pouvez vous connecter sur IRC (freenode), canal #ubuntu-uds-plenary. Voici un lien qui pourra être utile pour ceux peu familier avec cette technologie. Si certains sont très actifs, nous pourrons même les inviter dans le hangout pour poursuivre la discussion.

Rendez-vous donc ce soir, à 21h!

mercredi, août 14 2013

Release early, release often, release every 4h!

It's been a long time I didn't talk about our daily release process on this blog.

For those who are you not aware about it, this is what enables us to release continuously most of the components we, as the ubuntu community, are upstream for, to get into the baseline.

Sometimes, releases can feel like pushing huge rocks

Historic

Some quick stats since the system is in place (nearly since last December for full production):

  • we are releasing as of now 245 components to distro every day (it means, everytime a meaningfull change is in any of those 245 components, we will try to release it). Those are segregated in 22 stacks.
  • this created 309 uploads in raring
  • we are now at near 800 uploads in saucy (and the exact number is changing hour after hour)

Those numbers are even without counting feature branches (temporary forks of a set of branches) and SRUs.

Getting further

However, seeing the number of transitions we have everyday, it seems that we needed to be even faster. I was challenged by Rick and Alexander to speed up the process, even if it was always possible to run manually some part of it[1]. I'm happy to announce that now, we are capable of releasing every 4h to distro! Time for a branch proposed to trunk to the distro is drastically reduced thanks to this. But we still took great care to keep the same safety net though with tests running, ABI handling, and not pushing every commit to distro.

This new process was in beta since last Thursday and now that everyone is briefed about it, it's time for this to get in official production time!

Modification to the process

For this to succeed, some modifications to the whole process have been put in place:

  • now that we release every 4 hours, it means we need a production-like team always looking at the results. That was put in place with the ~ubuntu-unity team, and the schedule is publically available here.
  • the consequence is that we have no more "stack ownership", everyone in the team is responsible for the whole set now.
  • it means as well that upstream now have a window of 4 hours before the "tick" (look at the cross on the schedule) to push stuff in different trunks in a coherence piece rather than once a day, before 00 UTC. It's the natural pressure between speed versus safety for big transitions.
  • better communication and tracking were needed (especially as the production is looked after by different people along the day). We needed as well to ensure everything is available for upstreams to know where their code is, what bugs affects them and so on… So now, everytime there is an issue, a bug is opened, upstream is pinged about it and we write about those on that tab. We escalate after 3 days if nothing is fixed by then.
  • we will reduce as much as possible "manual on demand rebuild" as the next run will fix it.

Also, I wanted that to not become a burden for our team, so some technical changes have been put in place:

  • not relying anymore on the jenkins lock system as the workflow is way more complex than what jenkins can handle itself.
  • possible to publish the previous run until the new stack started to build (but still possible even if the stack started to wait).
  • if a stack B is blocked on stack A because A is in manual publishing mode (packaging changes for instance), forcing the publication of A will try to republish B if it was tested against this version of A. (also, having that scaling up and cascading as needed). So less manual publication needed and push button work \o/
  • Some additional "force rebuild mode" which can retrigger automatically some components to rebuild if we know that it's building against a component which doesn't ensure any ABI stability (but only rebuild if that component was renewed).
  • ensure as well that we can't deadlock in jenkins (having hundreds of jobs running every 4h).
  • the dependency and order between stacks are not relying anymore on any scheduling, it's all computed and ordered properly now (thanks Jean-Baptiste to have help on the last two items).

Final words

Since the beginning of the week, in addition to seeing way more speed up for delivering work to the baseline, we also have seen the side benefit that if everybody is looking at the issues more regularly, there are less coordination work to do and each tick is less work in the end. Now, I'm counting on the ~ubuntu-unity team to keep that up and looking at the production throughout the day. Thanks guys! :)

I always keep in mind the motto "when something is hard, let's do it more often". I think we apply that one quite well. :)

Note

[1] what some upstreams were used to ask us

jeudi, mars 7 2013

Followup UDS session on application lifecycle

I hope that everyone enjoyed the new virtual UDS format as we did. :)

However, don't despair, it's not really *over* yet! The discussion around the application lifecycle[1] was covering a too large spectrum and we didn't get time to properly finish discussing it. As we didn't want to run over other sessions, we decided to reschedule it just after UDS.

The new follow-up session "Application model: lifecycle" will happen this Friday, March 8, 16:00 – 16:25 UTC. It will be a hangout on air and people will follow up questions on #ubuntu-devel, IRC, freenode. We'll try to set up with the community an ubuntu on air session so that links are easier to find (this blog post will be updated with exact information near the event time).

Happy post vUDS session!

Note

[1] You can rewatch the video session here as it was live, thanks to the hangout on air!

mardi, février 5 2013

Unity: release early, release often… release daily! (part 5 and conclusion)

This post is part of the Unity daily release process blog post suite.

After a week to let people ask questions about the daily release process, I guess it's time to catch up and conclude this serie with a FAQ and some thoughts for the future.

FAQ

The FAQ is divided in multiple sequences depending on your role in the development of ubuntu, with the hope that you will be able to find what you are looking for quicker this way. If you have any question that isn't addressed below, please ping didrocks on IRC freenode (#ubuntu-unity) and I'll complete this page.

Upstream developers

What's needed to be done when proposing a branch or reviewing?

As discussed in this part, when proposing a branch or reviewing a peer's work, multiple rules have been established. The developers needs to:

  • Design needs to acknowledge the change if there is any visual change involved
  • Both the developer, reviewer and the integration team ensure that ubuntu processes are followed (UI Freeze/Feature Freeze for instance). If exceptions are required, they check before approving for merging that they are acknowledged by different parts. The integration team can help smooth to have this happened, but the requests should emerge from developers.
  • Relevant bugs are attached to the merge proposal. This is useful for tracking what changed and for generating the changelog as we'll see in the next part
  • Review of new/modified tests (for existence)
  • They ensure that it builds and unit tests are passing automated
  • Another important one, especially when you refactor, is to ensure that integration tests are still passing
  • Review of the change itself by another peer contributor
  • If the change seems important enough to have a bug report linked, ensure that the merge request is linked to a bug report (and you will get all the praise in debian/changelog as well!)
  • If packaging changes are needed, ping the integration team so that they acknowledge them and ensure the packaging changes are part of the merge proposal.

When you approve a change, do not forget to check that there is a commit message in launchpad provided and to set the global status to "approved".

When do changes need to land in a coherent piece?

Some changes can happen and touch various components (even across different stacks). To avoid loosing a daily release because only half of transition landed in some upstream projects (and so, tests will fail because they catch this issues, isn't it? ;)), it's asked to ensure that all transitions are merged by 00 UTC. Then, you can start approving again transition starting from 06 UTC. We can try to make this window shorter in the future, but let's see first how this work in practice. For any big transition, please coordinate with the ubuntu-unity team team so that they can give a hand.

I have a change involving packaging modifications

As told in a previous section, just ask for the ubuntu-unity team to do some review or assisting in doing those changes.

I want to build the package locally myself

If you have done some packaging changes, or added a C symbol, you maybe want to ensure that it's building fine. Quite simple, just go into your branch directory and run $ bzr bd (from the bzr-builddeb package). This will build your package on your machine using the same flags and checks than on the builders. No need to autoreconf if you are using autotools, everything is handled for you. You will get warned as well if some installed files are missing. ;)

I need to break an API/ABI

The first question is: "are you sure?". Remember that API changes are a PITA for all third parties, in particular if your API is public. Retro-compatibility is key. And yes, your dbus interface should be considered as part of your API!

If you still want to do that, ensure that you bump your soname if the ABI is broken, then, multiple packaging changes are needed. Consequently, ensure to ping your lovely ubuntu-unity team to request assistance. The transition should happen (if no retrocompatibility is provided) within the previously mentionned timeframe to have everything landing in coherence.

You also need to ensure that for all components build-depending on the API you changed, you bumped the associated build dependency version in debian/control.

To what version should I bump the build dependency version?

Let's say you added an API to package A. B is depending on A and you want to start using this new awesome API. What version should be used in debian/control to ensure I'm building against the new A?

This is quite easy, the build-dependencies in debian/control should look like: A >= 6.0daily13.03.01 (eventually ended by -0ubuntuX). Strip the end -0ubuntuX and append then bzr<rev_where_you_added_the_api_in_trunk>. For instance, let's imagine you added the API in rev 42 in trunk, so the build-dep will change from:

A >= 6.0daily13.03.01

to:

A >= 6.0daily13.03.01bzr42

This is to ensure the upstream merger will be able to take it.

If no "daily" string is provided in the dependency you add, please check with the ubuntu-unity team.

I'm exposing a new C symbols in my library, it seems that some packaging changes are needed…

Debian packages have a debian/<packagename>.symbols file which lists all exposed symbols from your library (we only do that for C libraries as C++ does mangle the name per architecture). When you try to build the package containing a new symbol, you will see something like:

--- debian/libdee-1.0-4.symbols (libdee-1.0-4_1.0.14-0ubuntu2_amd64)

+++ dpkg-gensymbolsvPEBau 2013-02-05 09:43:28.529715528 +0100

-4,6 +4,7

dee_analyzer_collate_cmp@Base 0.5.22-1ubuntu1

dee_analyzer_collate_cmp_func@Base 0.5.22-1ubuntu1

dee_analyzer_collate_key@Base 0.5.22-1ubuntu1

+ dee_analyzer_get_type@Base 1.0.14-0ubuntu2

dee_analyzer_new@Base 0.5.22-1ubuntu1

dee_analyzer_tokenize@Base 0.5.22-1ubuntu1

dee_client_get_type@Base 1.0.0

The diff shows that debian/libdee-1.0-4.symbols doesn't list the exposed dee_analyzer_get_type new symbol. You see a version number next to it, corresponding to the current package version. The issue is that you don't know what will be the package version in the future when the next daily release will happen (even if you can infer), what to put then?

The answer is easier than you might think. As explained in the corresponding section, the daily release bot will know what version is needed, so you just need to give a hint that the new symbol is there for the upstream merger to pass.

For than, just include the symbol, with 0replaceme[1] as the version for the branch you will propose as a merge request. You will thus set in your symbols file:

dee_analyzer_collate_cmp@Base 0.5.22-1ubuntu1

dee_analyzer_collate_cmp_func@Base 0.5.22-1ubuntu1

dee_analyzer_collate_key@Base 0.5.22-1ubuntu1

dee_analyzer_get_type@Base 0replaceme

dee_analyzer_new@Base 0.5.22-1ubuntu1

dee_analyzer_tokenize@Base 0.5.22-1ubuntu1

dee_client_get_type@Base 1.0.0

The dee_analyzer_get_type@Base 0replaceme line will be replace with the exact version we release in the next daily release.

My name deserves to be in the changelog!

I'm sure you want the world to know what great modifications you have introduced. To get this praise (and blame ;)), we strongly advise you to link your change to a bug. This can be done in multiple ways, like link bugs to a merge proposal before it's approved (you link a branch to the bug report in launchpad), or use "bzr commit --fixes lp:XXXX" so that automatically links it for you when proposing the merge for reviewing, or put in a commit message something like "this fixes bug…". This will avoid the changelog to have only empty content.

I invite you to read "Why not including all commits in debian/changelog?" from this section to understand why we don't include everything, but only bug reports title in the changelog.

Ubuntu Maintainers

I want to upload a change to a package under daily release process

Multiple possibilities there.

The change itself can wait next daily release

If it's an upstream change, we strongly advise you to follow the same procedure and rules than the upstream developers, meaning, proposing a merge request and so on… bzr lp-propose and bzr branch lp:<source_package_name> are regularly and automatically adjusted as part of the daily release process to point to the right branches so that you can find them easily. Vcs-Bzr in debian/control should point to the right location as well.

Then, just refer to the upstream merging guidelines above. Once approved, this change will be in the next daily release (if tests pass).

An urgent change is needed

If the change needs to be done right away, but can wait for a full testing round (~2 hours), you can:

  • Propose your merge request
  • Ping someone from the ubuntu-unity team who will ensure the branch is reviewing in priority and rerun a daily release manually including your change. That way, your change will get the same quality reviews and tests checking than anything else entering the distro.
It's really urgent

You can upload right away your change then, the next daily will be blocked for that component though (and only for that component) until your change reaches upstream. So please, be a good citizen and avoid more churn in proposing your change back to the upstream repo (including changelog), pinging the ubuntu-unity team preferably.

I'm making a change, should I provide anything in debian/changelog?

Not really, as mentioned earlier, if you link to a bug or mention in a commit message a bug number in your branch before proposing for review, this will be taken into account and debian/changelog will be populated automatically with your name as part of the next daily release. You can still provide it manually and the daily release will ensure there is no duplication if it was already mentionned.

If you provide it manually, just: dch -i and ensure you have UNRELEASED for an unreleased version to ubuntu (without direct upload).

ubuntu-unity team

Note that in all following commands, if -r <release> is not provided, "head" is assumed. Also, ensure you have your credentials working and be connected to the VPN (the QA team is responsible for providing those jenkins credentials).

Where is the jenkins start page?

There it is.

Forcing a stack publication

After reviewing and ensuring that we can release a stack (see section about manual publication causes, like packaging changes to review or upstream stack failing/in manual mode). Then run:

$ cu2d-run -P <stack_name> -r <release>

example for indicators, head release:

$ cu2d-run -P indicators

Remember that the master "head" job is not rerun, so it will stay in its current state (unstable/yellow). The publish job should though goes from unstable/yellow to green if everything went well.

Rerun a full stack daily release, rebuilding everything

$ cu2d-run -R <stack_name> -r <release>

Rerun a partial stack daily release, rebuilding only some components

$ cu2d-run -R <stack_name> -r <release> <components>

/!\ this will keep the state and take into account (failure to build state for instance) and the binaries packages for the other components which were part of this daily release, but not rebuilt.

for instance, keeping dee and libunity, but rebuilding compiz and unity from the unity, head release stack:

$ cu2d-run -R unity compiz unity

The integration tests will still take everything we have (so including dee and libunity) at the latest stack with the new trunk content for compiz and unity when we relaunched this command.

Rerun integration tests, but taking all ubuntu-unity ppa's content (like transition needing publication of the indicator and unity stacks):

$ cu2d-run -R <stack_name> -r <release> --check-with-whole-ppa

This command doesn't rebuild anything, just run the "check" step, but with the whole ppa content (basically doing a dist-upgrading instead of selecting only desired components from this stack). This is handy when for instance a transition involved the indicator and unity stack. So the indicator integration tests failed (we don't have the latest unity stack built yet). Unity built and tests pass. To be able to release the indicator stack as well as the unity stack, we need the indicator stack to publish, for that, we relaunch only the indicator tests, but with the whole ppa content, meaning last Unity.

In that case, the publish for the indicators will be manual (you will need to force the publication) and as well, you should ensure that you will publish the unity stack at the same time as well to have everything in coherence copied to the -proposed pocket. More information on the stack dependency in this section.

Adding/removing components to a stack

The stack files are living in jenkins/etc/. This yaml file structure should be straightforward. Note that the tuple component: branch_location is optional (it will be set to lp:component if not provided). So add/change what you need in those file.

Be aware that all running jobs are stopped for this stack.

$ cu2d-update-stack -U <path_to_stack_file>

This will reset/reconfigure as well all branches, add -S if you don't have access to them (but ensure someone will configure them still at least once)

Different level of failures acceptance.

We do accept some integration tests failing (no unit tests should fail though as ran as part of package building). The different level of those triggers are defined in a file like jenkins/etc/autopilot.rc. Those are different level of failures, regressions, skipped and removed tests per stack. The real file depends on stackname and are found in $jenkins/cu2d/<stack_name>.autopilotrc as described in the -check template.

Monitoring upstream uploads and merges

Remember to regularly look at the -changes mailing list to pick if a manual upload has been done and port the changes back (if not proposed by the ubuntu developer making the upload) to the upstream branch. The component which had a manual upload is ignored for daily release until this backport is done, meaning that no new commits will be published until then (you can see the prepare job for that component will turn unstable).

Also, ensure as you are watching all upstream merges to follow our acceptance criterias that the "latestsnapshot" branches generated automatically by the daily release process has been merged successfully by the upstream merger.

Boostrapping a new component for daily release

A lot of information is available on this wiki page.

Conclusion

Phew! This was a long serie of blog post. I hoped that you appreciated to have a deeper look of what daily release means, how this work, and also that the remaining questions have been answered by the above FAQ. I've just ported all those information on the https://wiki.ubuntu.com/DailyRelease.

Of course, this is not the end of the road, I have my personal Santa Claus wishlist like UTAH being more stable (this is under work), the daily image provisionning with success more often than now, refactorisation work taking integration tests into account, less Xorg crashes when completing those tests… This is currently adding a little bit of overhead as of course, any of this will reject the whole test suite and thus the whole stack. We prefer to always be on the safe side and not accepting something in an unknown state!

Also, on the upstream stack and daily release process as well, there are rooms for improvments, like having less upstream integration tests failing (despite awesome progress already there), I wish I can soon turn that down to 3% of accepted failure at most for unity instead of the current 7% one. Dedicated integrations tests for indicators would be awesome as well (stealing the unity ones for now and only running those involving indicators, but of course, not everything is covered). Having also some oif, webapps and wecreds integration tests is one of my main priority, and we are working with the various upstream developers to get this done ASAP so that we have a better safety net than today. On the daily release process itself, maybe freezing all trunks in a go would be a nice improvement, more tests for the system itself as well is under work and my personal goal to reach 100% test coverage for the scripts before making any additional changes. Finally, some thoughts for later but a dashboard would enable to have a higher view than just using jenkins directly for reporting, to get a quicker view on all stack statuses.

Of course, I'm sure that we'll experience some bumpy roads with this daily release process through the cycle, but I'm confident we'll overcome them and I am really enthousiast about our first releases and how smoothly they went until now (we didn't break the distro… yet :p[2]. I'm sure this is a huge change for everyone and I want again give a big thank you to everyone involved into that effort, from the distro people to upstream and diverse contributors on the QA team. Changing to such a model isn't easy, it's not only having yet another process, but it's really a change of mindset in the way we deliver new features and components to Ubuntu. This might be frightening, but I guess it's the only way we have to ensure a systematic delivery, continuous and increasing quality while getting well a really short feedback loop to be able to react promptly to the changes we are introducing to our audience. I'm confident we are already collecting all the fruits of this effort and going to expand this more and more in the future.

For all those reasons, thanks again to all parties involved and let's have our daily shot of Unity!

I couldn't finish without a special thanks to Alan Pope for being a patient first reader, and fixing tediously various typos that I introduced into the first draft of those blog posts. ;)

Notes

[1] "0replacemepleasepleaseplease" or anything else appended to 0replaceme is accepted as well if you feel adventurous ;)

[2] fingers crossed, touching wood and all voodoo magic required

vendredi, janvier 25 2013

Unity: release early, release often… release daily! (part 4)

This post is part of the Unity daily release process blog post suite.

You hopefully recovered from your migraine on reading yesterday's blog post on the insight of daily release and are hungry for more. What's? That's not it? Well, mostly, but we purposely dismissed one of the biggest point and consequences on having stacks: they depends on each other!

Illustrating the problem

Let's say that Mr T. has an awesome patch for the indicator stack, but this one needs as well some changes in Unity and is not retro-compatible. Ok, indicators integration tests should fail, isn't it? as long as we don't have the latest version of Unity rebuilt in the ppa… But then, the indicator stack is not published (because of integration tests failing), but Unity, with its integration tests will pass as they are run against the latest indicator stack built in the same ppa. Consequently, we may end up publishing only half of what's needed (the Unity stack, without the indicator requirement).

Another possible case: let's imagine that yesterday's daily-build failed and so we have a new unity staging in the ppa (containing the indicator-compatible code), so the next day, the indicator integration tests will pass (as it's installing Unity from the ppa), but will only publish the indicator components, not the Unity ones. Bad bad bad :-)

We clearly see we need to declare dependencies between stacks and handle them smartly.

What's the stack dependency as of today?

Well, a diagram is worth 1 000 words:

Stack dependencies

We can clearly see here that stacks are depending on each other, this is specified in the stack configuration file, each stack telling on what other they depend on.

This has multiple consequences.

Waiting period

First, a stack will start building only once all each stack it depends on finished to build. This enables us to ensure we are in a predictable state and not in a in-between with half of the components building against the old upstream stack and the other half against the new upstream stack because it finished to build in the same ppa.

The previous diagram needs to be then modified for the unity stack with this "wait" optional new step. This step is only deployed with cu2d-update-stack if we declared some dependent stacks.

Daily Release, jenkins jobs with wait optional step

The wait script is really simple to achieve that task. However, waiting is not enough as we've seen above, we need to take the test result into consideration, which we'll do later on during the publisher step.

Running integration tests in isolation

Ok, that's answering some part of the consistency issue, however, imagine now the second case we mentionned where yesterday's daily build failed, it means that if we take the whole ppa content, we'll end up testing the new indicator integration with a version of Unity which isn't the distro one, nor the one we'll publish later on. We need to ensure we only tests what will make sense.

The integration jobs are then not taking the whole ppa content, but only the list of components generated by the current stack. It means that we are explicitely only installing components from this stack and nothing else. Then, we log the result of the install command and filters against it. If a soname change for instance (like libbamf going from 3.0 to 3.1), ending up in a package renaming will grab a new Unity to be able to run, we'll then bail out.

Here is a real example where we can find here the list of packages we are expecting to install, then here is the list of what happened when we asked to install only those packages. The filtering happening as the first test will then exit the job in failure, meaning we can't publish that stack.

So we know if the we are ensuring we know the retro-compatibility status this way. If retro-compatibility is achieved, the tests are passing right away, and so the stack is published regardless other stacks, as we tested against Unity distro's version. This ensure we are not regressing anything then. In other case when retro-compatibility seems to fails, as in the previous example, this means that we would eventually have to publish both stacks at the same time, which is what we can do.

Ensuring that the leaf stack is not published

In a case like the previous one, the Unity integration tests will run against the latest indicator stack, and then they pass (having latest everything), but wait! We shouldn't publish without the indicator components (and those failed, we are not even sure the fixes in Unity will make them pass and we haven't tested againt the whole PPA).

Here comes the second "manual publishing" case from the publisher script I teased about yesterday. If a stack we depend on didn't finish with success, the current stack is put in manual publication. The xml artefacts giving the reason is quite clear:

indicators-head failed to build. Possible causes are:

* the stack really didn't build/can be prepared at all.

* the stack have integration tests not working with this previous stack.

What's need to be done:

* The integration tests for indicators-head may be rerolled with current dependent stack. If they works, both stacks should be published at the same time.

* If we only want to publish this stack, ensure as the integration tests were maybe run from a build against indicators-head, that we can publish the current stack only safely.

So from there, we can be either sure that the upstream stack has no relationship with the current downstream one, and so, people with credentials can force a manual publishing with cu2d-run as indicated in the previous blog post. However, if there is a transition impacting both stacks, the cleverest way is to relaunch the upstream stack tests with the whole ppa to validate that hypothesis.

For that, cu2d-run has another mode, entitled "check with whole ppa". This option will rerun a daily-release, but won't prepare/rebuild any component. It will though still takes their build status into account for the global status. The only difference is that the integration tests will take into account this time the whole ppa (and so no more filtering on what we install if it's part of the current stack or not), doing a classic dist-upgrade. This way, we validate with "all latest" that the tests are passing.

Note that even if tests pass, as we'll probably need to publish at the same time than the dependent stack, we'll automatically pass in manual publishing mode. Then, once all resolved, we can publish all stacks we want or need with cu2d-run.

Manual publish cascades to manual publish

Another case when a dependent stack can fallback to manual publishing mode doesn't necessarily imply tests failing. Remember the other case when we fallback in manual publishing mode? Exactly, packaging changes! As the dependent stack is built against the latest upstream stack, and as it's what we tested, we probably need to ensure both are published (or force a manual publishing on only one if we are confident that there is no interleaved depends in the current change).

Here is what the artefacts will provide as infos on that case:

indicators-head is in manually publish mode. Possible causes are:

* Some part of the stack has packaging changes

* This stack is depending on another stack not being published

What's need to be done:

* The other stack can be published and we want to publish both stacks at the same time.

* If we only want to publish this stack, ensure as the integration tests were run from a build against indicators-head, that we can publish the current stack only safely.

So, here, same solutions than in the previous case, with cu2d-run, we either publish both stack at the same time (because the packaging changes are safe and everything is according to the rules) or just the leaf one if we are confident there is no impact.

Note that we do handle as well if something unexpected happened on a stack and that we can't know its status.

In a nutshell

Stacks don't live in their own silos, they are high dependency behavior between them which are ensured by the fact that they live and are built against each other in the same ppa. We as well ensure the retro-compatibility mode, which is what most of daily release will be, than only publishing some stacks. We also handle that way transitions impacting multiple stacks and can publish everything in one shot.

That's it for the core of the daily release process (phew, isn't it?). Of course this is largely simplified and we have a lot of corner cases I didn't discuss about, but this should give enough high coverage on how the general flow is working out. I'll wrap it up in a week with some kind of FAQ from the most common questions that are still maybe pending (do not hesitate to comment here) and give this time to collect feedbacks. I will publish this FAQ then and at the same time will draw a conclusion from there. See you in a week. ;)

- page 1 de 3