15:12:46 <itchka_> #startmeeting
15:12:46 <chwido> Woof! Let's start the meeting. It's Wed Aug 17 15:12:46 2016 UTC. The chair is itchka_. Information about me at https://wiki.openmandriva.org/en/Chwido.
15:12:46 <chwido> Useful Commands: #action #agreed #help #info #idea #link #topic.
15:12:46 <chwido> You should add extra chair(s) just in case, with #chair nick.
15:12:46 <chwido> Have a good meeting, don't bark too much!
15:12:56 <itchka_> #item 1
15:13:24 <itchka_> #topic Implement Rolling elease
15:13:41 <itchka_> Release strategey
15:13:59 <Pharaoh_Atem> is the tooling capable of handling a rolling release system at all?
15:14:12 <Pharaoh_Atem> one of the huge problems with rolling release is dependency drift
15:14:30 <Pharaoh_Atem> openSUSE solved this problem by tracking forward and reverse dependencies in OBS and just rebuilding things automatically
15:14:44 <Pharaoh_Atem> Fedora is working on a similar thing for Koji called Koschei
15:14:59 <Pharaoh_Atem> currently it works in scratch builds only, but it will eventually get integrated into the main system
15:15:36 <bero> would be good to have...
15:15:50 <Pharaoh_Atem> for example: https://apps.fedoraproject.org/koschei/package/pypy
15:15:51 <bero> but for now it's a manual process and that doesn't really impact rolling vs. old-style release
15:15:59 <bero> it has to be done manually either way
15:16:19 <Pharaoh_Atem> but if you actually decide to move forward with making a rolling variant, it *will* be a problem
15:16:23 <itchka_> Pharaoh_Atem: I and probably other would welcome some kind of clarification of the term "Dependency Drift" if this can be done succinctly
15:16:58 <Pharaoh_Atem> itchka_: dependency drift is essentially when you update some components but you don't update all the things that depend on those components concurrently
15:17:17 <Pharaoh_Atem> essentially, depending on the library/program/whatever, the drift could be subtle or actually break things
15:17:22 <bero> Pharaoh_Atem: not more than before - we fully intend to keep cooker as it is now, we just want to add a sanity-checked tree (that will get updated once the dependency drift is resolved)
15:17:23 <Pharaoh_Atem> for example, ffmpeg and mpv
15:17:55 <Pharaoh_Atem> ffmpeg hasn't bumped its sover in a while, but it makes changes that break the ABI subtly all the time
15:18:18 <Pharaoh_Atem> in OBS, updating ffmpeg automatically rebuilds and pushes a new mpv
15:18:34 <Pharaoh_Atem> this eliminates the dependency drift and any issues caused by that
15:19:17 <Pharaoh_Atem> Koschei and OBS use the same technology to implement this (libsolv)
15:19:36 <bero> and we're currently seeing some breakage in cooker from a harder version of dependency drift (python 3.4 -> 3.5)
15:19:37 <Pharaoh_Atem> OBS does it through perl-BSSolv, while Koschei does it through python-hawkey
15:19:58 <Pharaoh_Atem> bero: that would be caught and automatically fixed as well with dependency tracking
15:20:14 <bero> yes, it would be really nice to have that
15:20:21 <Pharaoh_Atem> so if ABF (your current build system) integrated with hawkey to provide dependency tracking, it can handle this case for you
15:20:29 <Pharaoh_Atem> at least at the minimum tell you what you need to rebuild
15:20:38 <bero> fedya: fedya_: HisShadow: avokhmin_: ^^^^^ ABF feature request ;)
15:21:39 <Pharaoh_Atem> hawkey and libhif libraries are already available in omdv, so you can use either one for this
15:22:31 <itchka_> But say we can't do this immediately are there actually many cases where this is actually a problem? If we can identify them then we can manage the entire thing through build-lists for the time being.
15:23:01 <Pharaoh_Atem> things like ffmpeg, python, perl, chromium/v8, nodejs, ruby, etc. tend to be a problem
15:23:33 <Pharaoh_Atem> especially golang, rust, nodejs, etc
15:23:39 <Pharaoh_Atem> since they have no ABI guarantees at all
15:24:18 <HisShadow> I don't know if people still install ruby and node into there system since there is rvm and nvm
15:24:21 <bero> itchka_: we can manage it with build lists for the time being - QA just has to be a little careful in approving library updates from cooker to the rolling tree
15:24:45 <Pharaoh_Atem> HisShadow: if they're going to run a huge application, they usually do
15:25:38 <bero> and they should... I'd rather trust my distribution than pulling in random stuff that was tested on Ubuntu only and might not even compile
15:25:42 <Pharaoh_Atem> fedya: HisShadow: avokhmin_: I'm not sure how the architecture is done in the Python/Ruby split in ABF, but if this is something the python side can handle, I'd recommend using python-hawkey
15:26:53 <Pharaoh_Atem> otherwise, you might want to implement ruby bindings for hawkey to do it
15:27:01 <itchka_> bero: Currently there is much we can do to QA libraries except to install them. I've been wanting to QA them for some time and have some ideas which I put forward for the semester of code. Thos eideas would be a lot of work to implement and probably beyond my slkill set. So we are going to need some sort of QA early warning system.
15:27:40 <bero> right
15:27:48 <Pharaoh_Atem> at the *very minimum*, tracking dependencies can give you warnings of this issue that can be acted on
15:28:46 <Pharaoh_Atem> since I doubt you guys expose a koji hub interface, I don't think you could use Koschei as-is: https://github.com/msimacek/koschei
15:28:55 <Pharaoh_Atem> but you could probably integrate it into your platform
15:29:17 <Pharaoh_Atem> on the Mageia side, we're looking into using Koji+Koschei for exactly this reason
15:29:59 <bero> yes, should be possible to just make it dump a list of packages and then feed it into abf's build list interface
15:30:17 <Pharaoh_Atem> we don't necessarily want the automatic building and pushing yet, but having a system that actively tracks in real-time the state of the repos and can test with scratch builds when drifts occur
15:30:19 <itchka_> So from a QA point of view if a library is updated all the applications that have that lib as a dependency need to be updated as well.
15:30:23 <Pharaoh_Atem> helps a lot
15:30:42 <Pharaoh_Atem> Koschei does that through Koji's scratch build capability using mock
15:30:54 <bero> itchka_: exactly -- it's just a matter of making sure all the bits that need to go together go into the sane version at the same time
15:31:17 <itchka_> So what happens aboubt libc, libm and the like does that mean a mass rebuild?
15:31:51 <Pharaoh_Atem> essentially, yes
15:32:04 <Pharaoh_Atem> if gcc, libc, libstdc++, et al get updated, it triggers a mass rebuild
15:32:10 <itchka_> Holy cow
15:32:17 <Pharaoh_Atem> it works surprisingly well
15:32:17 <bero> Ideally we'd have a mass rebuild for libc or compiler changes as a test. Realistically, they rarely break the ABI in unexpected ways, so for now we can get away with putting those on an ignore list
15:34:11 <itchka_> We are going to need a lot more builders if we are to do regular mass builds.
15:34:31 <bero> that's why I'm saying just blacklist glibc for now
15:34:35 <Pharaoh_Atem> this is why rolling release is really hard
15:34:44 <bero> besides we don't touch glibc all that often
15:34:46 <Pharaoh_Atem> doing rolling release right is a fundamentally difficult challenge
15:37:04 <itchka_> The problem I see with the current approaches is that you have to assume that the abi has been broken so you may recompile many packages uncecessarily.
15:39:38 <Pharaoh_Atem> there are ways to change that assumption
15:39:43 <Pharaoh_Atem> you could do it on symbol tracking
15:40:17 <Pharaoh_Atem> for example, if a runtime requirement has disappeared but the buildrequires are still satisfiable, you could rebuild only in that case
15:40:23 <Pharaoh_Atem> that's the way Koschei works right now, I believe
15:40:32 <Pharaoh_Atem> OBS is more "greedy" and builds on all updates by default
15:42:01 <Pharaoh_Atem> in the former scenario, a glibc update would not trigger a mass rebuild unless it dropped a symbol that broke everything
15:46:00 <itchka_> Surely though these brealkages are not limited to missing symbols you indicated that subtle changes in the abi might cause things not to work indicating that the sysmbol exists but the implementation has changed.
15:47:32 <itchka_> So it would seem that neither strategy will catch all cases
15:51:19 <bero> Pretty much every library considers a subtle ABI change a bug, so that doesn't happen all that often
15:51:22 <bero> but it does happen
15:53:57 <Pharaoh_Atem> yeah
15:54:14 <Pharaoh_Atem> you have to pick a strategy *shrugs*
15:55:30 <itchka_> Ok so missing symbols is the main issue which will presumably show up as soon as the program is run or can it be more subtle than that in that the failure only occurs when the missing library function is called.
15:56:10 <OnlyHuman> sounds like may be worth investing time in it, could save a lot of greif :
15:56:40 <bero> itchka_: usually that would show up as soon as a program is run - but it can also lead to a runtime failure in case the symbol is accessed through dlopen instead of "normal" linkage. (That's rare, but of course it's no "this is never going to happen" thing)
15:58:01 <jbj> ROSA’s update-checker is pretty damn good at detecting ABI associated build breakage. fully automating the decision criteria (beyond post-mortem forensics) is rather tricky
15:58:29 <itchka_> bero: So what that tells me is that if we throughly test updated applicatyions in QA after library updates we can be fairly sure taht we would get all but the really subtle cases.
15:59:03 <itchka_> If of course we knew which apps called thos elibs.
15:59:24 <bero> jbj: agreed -- we should try to get ROSA's update checker to detect any potential issues
15:59:37 <bero> itchka_: we do know that -- urpmq --whatrequires packagename
16:00:55 <jbj> bero: that graph search ripples out rather fast … and truncating the graph doesn’t solve the engineering problem
16:01:11 <Pharaoh_Atem> it gets really crazy really quickly
16:01:11 <itchka_> Ok so it's doable manually with a little thought and preparation.
16:02:04 <itchka_> the necessary info could be supplied as part of the Kahinah QA prog
16:02:18 <jbj> KISS: do speculative rebuilds of nearest neighbors before inflicting a build on repo lusers
16:03:04 <OnlyHuman> sounds good :)
16:04:22 <jbj> i.e. establish mandatory acceptance criteria before sending packages to mirrors. this really isn’t any different than “broken dependency” checks
16:04:44 <bero> I think that's the way to go -- do manual checks at first, and then automate stuff that can be automated over time
16:06:03 * ben79_ OK now I can vore twice
16:06:27 * ben79_ OK now I can vote twice
16:13:41 <itchka_> What other tools do we need to implement over and above this "Dependency Creep" control?
16:15:38 <bero> probably an abf feature that makes sure a list of packages get published all at once so we don't get complaints from users trying to urpmi --auto-update in the middle of something and its reverse dependencies being published
16:18:55 <itchka_> Yes that sounds sensible I think that could probably be achieved via the build list interface. If you are using a build list it is assumed that there are package interdependencies so the whole lot must be published together. The same would appl;y to kahinah
16:20:04 <bero> yes... In kahinah, we probably need an extra thing to mark dependencies to make sure they all get approved at the same time
16:21:12 <fedya> hi
16:21:15 <itchka_> Could we not manage this the task type rpms? like task-kde etc?
16:23:08 <itchka_> Hi fedy
16:23:14 <itchka_> fedya
16:28:34 <jbj> bero: chained build
16:29:38 <bero> itchka_: not really, we can't have a task-everything-that-depends-on-ffmpeg or so (would be messy for users and also there would be too many task-*s for just about every package)
16:31:24 <itchka_> I don't quite mean it like that. The task-buildallthis.rpm wouldonly be local to abf and it would be "installed" into the repo so the actual task would be invisible to users.
16:32:15 <jbj> itchka: task package is one way to aggregate multi-pkg components … but the aggregation is only as good as the task package is … automating, say, a BDB upgrade quickly includes a significant no. of critically important packages. the better approach (to BDB-like) is an incremental approach: leave a compatibility layer and move pkg-by-pkg to new ABI. BDB is easy: gnome/kde upgrades are more complex.
16:32:36 <Pharaoh_Atem> exactly
16:34:00 <itchka_> BDB? Please enlighten
16:34:18 <jbj> Berkeley DB … lots of usage cases everywhere
16:37:16 <itchka_> Hmm ok not fully onm that wavelength yet but I have tuned to the basic signal
16:38:03 <jbj> ffmpeg (& libpng) changes affect lots of GUI … BDB is perhaps easier model because it affects core services/daemons
16:40:30 <jbj> devising a way to add version in path so that old and new can coexist without clobbering each other, and then choosing a single (and easily flipped) development version is what works for BDB. there are surely other similar examples, I just known BDB well.
16:41:52 <itchka_> I do not see these task rpms as accessible to the user they will simply be a wrapper for moving and archive of rpms through the build and quality systems. The user would not install these tasks they would see what they always see multiple rpms
16:42:03 <jbj> the other death trap is pooping out before the old version is totally eradicated.
16:42:48 <itchka_> I quite undestand what you are getting at now.
16:43:58 <bero> libpng already does something quite similar to BDB - versioned name (libpng16.so) and a symlink from libpng.so to libpng16.so so newly built stuff that uses -lpng will get the latest version while stuff explicitly needing 1.6.x can use -lpng16
16:44:51 <jbj> itchka_: you will likely do better with a “task repo” (i.e. a separate repository) than a “task package”. repo’s can be visually vetted, dependency chains are often unexpectedly snarly.  otoh, a separate repository requires a deep understanding of the dependency snarliness too (but its rather easy to add a package to a repository when needed).
16:50:35 <itchka_> That's pretty much what I was getting at, remember these rpms are not going to be directly published if we use the new scheme they will end up in a testing repo and will be released from there after they have been tested they will be released under as the 'repo'.rpm which will be installed into the main updates repos as individual rpms.
16:50:38 <jbj> the alternative to incremental transition is a “flag day” … a sharp cut-over with a planned escape hatch if you do not succeed. but “flag days” don’t really fit into a “rolling release” model
16:51:47 <itchka_> But hey I'm sure there are loads of ways to skin this particular cat.
16:52:23 <bero> I think a combination of incremental transition and flag day can work (usually going for incremental transition with flag day for some special cases that are expected to break just about everything)
16:53:23 <jbj> “access” and “release” are mechanistically equivalent … and essentially a “rawhide” model dog-fooding lusers needlessly. a pre-planned acceptance test (and bail-out) are needed for transparency so decisions can be made without stress.
16:56:20 <itchka_> Ok folks I think we are begining to digress. We need to focus on what we MUST do before we can implement the proposed rolling release model. Then I think we need to have a vote to officially accept it as the way forward.
16:57:17 <jbj> there’s much to be said for the feature plans used in Fedora
16:57:38 <itchka_> The main item for QA is the ability to release groups of packages from Kahinah.. Without this functionality we cannot really support the model.
16:58:22 <jbj> at least one knows what SHOULD be happening.
17:00:08 <itchka_> Secondly when libraries are published kahinah should provide a urpmi --whatrequires list to indicate which programs need testing to guard against dependency creep.
17:01:09 <jbj> QA also needs an explicit manifest and fwd plan. I once spent 3 months getting 2 libraries rebuilt to change one symbol: 2 QA guys lacked understanding of the RPM <-> SELINUX deadly embrace and were each loath (being conservative) to decide and tread on each other’s turf. total cluster f**k
17:01:18 <itchka_> bero: What are the requirements from the dev side?
17:03:46 <bero> itchka_: from dev side, essentially things remain at "throw stuff at cooker and if it breaks, fix it" -- we just need to add some way to make sure packages arrive in the kahinah queue and don't get forgotten about. We'll also need a way to tag a package as "known broken, don't bother to submit yet" (to submit later along with other packages that need to be built because of it)
17:04:43 <itchka_> we could give it a -50 votes :)
17:05:10 <jbj> bero: “known” is the blacklist … there’s often a gray list where no one knows or work is incomplete (you know this ;-)
17:05:51 <itchka_> Then wehen we release the lot we could use qa push to push all the packages with a particular list tag
17:06:02 <bero> jbj: right... but catching those is probably QA's work even more than dev's work
17:06:27 <itchka_> qa push would override any negative  voting
17:07:21 <bero> another thing we should have is a tool that automatically checks differences between cooker and rolling release - to make sure we don't forget about a change that has been made in cooker and for whatever reason didn't get to the rolling release
17:07:42 <itchka_> We already manage this situtions either by consulting with the package or by simple common-sense
17:07:55 <jbj> bero: instead of “throw at cooker”, I’d suggest “throw largfe incompatibilities” at a task repo for “access”. several (5-10) “task repos” of moderate duration/lifetime is more flexible than “throw at cooker”. but perhaps that is what you meant by “cooker” ;-)
17:07:57 <itchka_> package
17:08:01 <itchka_> packager
17:08:19 <itchka_> Getting tired been up since 5am
17:09:13 <jbj> mini-me cooker repositories (if the reference isn’t too obsure)
17:09:24 <bero> jbj: yes -- essentially a mix of throwing directly at cooker for small bits (e.g. libpng 1.16.4 -> libpng 1.16.5) and task repo for huge incompatible things is probably the way to go
17:09:59 <jbj> planning? transparency?
17:10:43 <bero> 3.1 planning is one of the next agenda items
17:11:49 <jbj> release planning is too coarse: decompose into features/tasks with per-cent completion … essentially the fedora model
17:12:42 <itchka_> Please let's do one thing at a time:)
17:15:36 <bero> agreed for some big tasks -- but we must make sure we don't lose too much flexibility from over-planning - we don't know what may happen outside our project that will affect what way we have to go. We need to find the right mix between proper planning and retaining flexibility.
17:15:47 <itchka_> We can't adopt the rolling release model until we have the tools to do it. I think we have identified the tools we need so now we need to work out how we are going to create them.
17:17:22 <jbj> bero: scrapping a feature by rejecting the task repo merge -> cooker seems flexible
17:20:32 <jbj> filling in a feature plan might seem “obvious” busy work for, say, next KDE release, but percent completion, as well as merge event dates help considerably imho. I know I look atFedora features when having WTF?!? moments in order to efficiently work around issues. jmho …
17:25:33 <itchka_> fedya: HisShadow: Simple question...is it possible to inregrate Kahinah into abf with the features that have been suggested?
17:31:06 <itchka_> Ok they don't seem to be about.
17:34:48 <itchka_> We have been meeting now for 2.5 hours I would suggest that we defere the other agenda items until the next meeting. I think we should have a vote as to whether to accept the rolling release model as described in TPG's document. If the model is accepted tehn we will need to deal with the tools issue promptly else items 2 and 3 of the agenda will not be going anywhere.
17:35:42 <itchka_> All those in favour of accepting the new rolling release model?
17:38:45 <itchka_> +1
17:39:09 <itchka_> #chair bero
17:39:09 <chwido> Current chairs: bero itchka_
17:40:01 <itchka_> I have to go and feed the troops I will be back later to do the share and close it if ti has not already been done.
17:40:39 <bero> +1
17:50:31 <OnlyHuman> +1
17:51:18 <OnlyHuman> chwido: bark efl
17:51:19 <chwido> refuses to bark at such a nice soul.
17:52:08 <jbj> +1 … “rolling” is the new hotness
17:52:49 <Pharaoh_Atem> ehh
17:52:59 <Pharaoh_Atem> -1
17:53:14 <Pharaoh_Atem> even though it's the hotness, it's also hard, and I'm not sure we can get everything together to make it work
17:53:44 <OnlyHuman> Keep Rollin', Rollin', Rollin'  Keep them dawwgies rollin',
17:54:51 <bero> It's low risk - we don't have to force 3.0 users on it
17:55:10 <OnlyHuman> well I'm a noob an have no idea about any of it, but sounded ok  :)
17:55:18 <bero> we can play around with it by having a rolling tree before 3.1 and make it official with 3.1 if it has worked out for those testing it before
18:02:46 <StatelessCat> 17T170945* itchka_ | Who is StatelessCat is he a friend for chwido? »» Hey, I'm someone curious after peoples told me that there is a binary distro wich use Clang. I was wondering and hoping (i'm not )
18:03:40 <StatelessCat> skilled about compilers, but I like clang. So people here answered me.
18:05:35 <ben79> /me Abstain
18:06:04 <ben79> Woops!
18:06:10 * ben79 abstain
18:06:39 <bero> StatelessCat: welcome aboard -- yes, we do use clang as our primary compiler -- 3.8.1 for the recently released 3.0, and our development tree has already moved on to the 3.9 branch
18:12:31 <ben79> User ? on Forum: https://forum.openmandriva.org/t/does-calamares-support-lvm2-volumes/632
19:26:47 <itchka_> Ok I'm going to close the meeting now
19:40:38 <itchka_> #share The topic of the meeting was a discussion and vote on the acceptance of the proposed Rolling Release strategy. The main issue pointed out by Pharoah Atem that there was an issue with it. There was a long discussion about "Creeping Dependencies" which is a situation that can occur in rolling releases where libraries get out of sync with applications. A number of methods were discussed that might be used to
19:40:40 <itchka_> manage this situation. It would be necessary to make some changes to Kahinah to manage dependent builds such as kde-libs.  Some testing would have to be performed on libraries (by using applications that call them) to ensure the the ABI has not been compromised. Finally a vote was takes as to whether to adopt the new model. The vote was 4 to 1 with one abstention. Some key members were not present so we may wish
19:40:42 <itchka_> to consider this a guide vote. The balance of the agenda was deferred until the next meeting.
19:41:05 <itchka_> #share The topic of the meeting was a discussion and vote on the acceptance of the proposed Rolling Release strategy. The main issue pointed out by Pharoah Atem that there was an issue with it. There was a long discussion about "Creeping Dependencies" which is a situation that can occur in rolling releases where libraries get out of sync with applications. A number of methods were discussed that might be used to
19:41:06 <itchka_> manage this situation. It would be necessary to make some changes to Kahinah to manage dependent builds such as kde-libs.  Some testing would have to be performed on libraries (by using applications that call them) to ensure the the ABI has not been compromised. Finally a vote was takes as to whether to adopt the new model. The vote was 4 to 1 with one abstention. Some key members were not present so we may wish
19:41:08 <itchka_> to consider this a guide vote. The balance of the agenda was deferred until the next meeting.
19:41:37 <itchka_> #endmeeting