11:24:50 <kashyap> #startmeeting
11:24:50 <zodbot> Meeting started Tue Jan  7 11:24:50 2014 UTC.  The chair is kashyap. Information about MeetBot at http://wiki.debian.org/MeetBot.
11:24:50 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
11:24:57 <kashyap> #meetingname RDO-test-day-JAN2014
11:24:57 <zodbot> The meeting name has been set to 'rdo-test-day-jan2014'
11:25:11 <mbourvin> panda: why would it need internet connection in order to install it locally?
11:25:30 <tshefi> giulivo, yep that's correct
11:27:15 <rlandy> mbourvin: I installed with packstack (all in one) yesterday - hit some workarounds along the way - but it was quicker. Today I am doing the same install over again and it is taking a *long* time
11:27:27 <rlandy> ie: it's not done yet
11:28:17 <mbourvin> rlandy: ok thanks, I'll wait a bit more
11:28:24 <ohochman> pixelb: can you check on : yum install http://rdo.fedorapeople.org/openstack-icehouse/rdo-release-icehouse.rpm'
11:28:46 <jistr> pixelb: it was just one occurrence, testing now: https://github.com/redhat-openstack/astapor/pull/95
11:29:44 <rlandy> mbourvin: yay - install just finished
11:30:16 <kashyap> ndipanov, However: after restart of OpenStack Compute service, still I don't see Compute node in $ nova-manage service list, and that's the fresh api.log : http://paste.fedoraproject.org/66362/09410513
11:30:24 <mbourvin> rlandy: good for you! mine is still running :(
11:30:28 <ohochman> kashyap: jistr: does it work for you  ?  '  yum install http://rdo.fedorapeople.org/openstack-icehouse/rdo-release-icehouse.rpm  '
11:30:53 <ndipanov> and compute.log kashyap ?
11:31:11 <kashyap> ohochman, I used: http://repos.fedorapeople.org/repos/openstack/openstack-icehouse/rdo-release-icehouse-1.noarch.rpm
11:31:29 <ohochman> kashyap: is it the same ?
11:31:40 <kashyap> ndipanov, Libvirt traces:
11:31:41 <kashyap> 2012-12-10 21:57:27.580 1356 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 76, in tworker
11:31:41 <kashyap> 2012-12-10 21:57:27.580 1356 TRACE nova.openstack.common.threadgroup     rv = meth(*args,**kwargs)
11:31:42 <kashyap> 2012-12-10 21:57:27.580 1356 TRACE nova.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3622, in baselineCPU
11:31:42 <kashyap> 2012-12-10 21:57:27.580 1356 TRACE nova.openstack.common.threadgroup     if ret is None: raise libvirtError ('virConnectBaselineCPU() failed', conn=self)
11:31:45 <kashyap> 2012-12-10 21:57:27.580 1356 TRACE nova.openstack.common.threadgroup libvirtError: internal error: CPU feature `avx' specified more than once
11:31:48 <kashyap> 2012-12-10 21:57:27.580 1356 TRACE nova.openstack.common.threadgroup
11:31:50 * kashyap uses pastebin. Sorry.
11:32:26 <nmagnezi> anyone managed to get ovs running?
11:32:45 <kashyap> ohochman, I haven't checked, but you can just do a "$ less foo.rpm" & can check the changelog, it should tell you.
11:33:00 <ohochman> ukalifon: yum install http://rdo.fedorapeople.org/openstack-icehouse/rdo-release-icehouse.rpm
11:34:01 <jistr> ohochman: the command did work for me, although i didn't start installing anything else after that, i'm on the package array issue now
11:34:17 <rlandy> mbourvin: on the upside, it seems to be working now - my previous install was a just an error-generator
11:34:44 <kashyap> nmagnezi, I have it running, : http://paste.openstack.org/show/60611/ ; Also, I stumbled across: https://bugzilla.redhat.com/show_bug.cgi?id=1049235
11:34:59 <kashyap> (for some definitions of "running" :-) )
11:36:02 <rlandy> ohochman: 'yum install http://rdo.fedorapeople.org/openstack-icehouse/rdo-release-icehouse.rpm' works for me so far
11:36:45 <yfried> was anyone able to get external network connectivity? (ping 8.8.8.8)
11:37:26 <yfried> or neutron-openvswitch-agent to come alive?
11:37:48 <yfried> I'm getting: abrtd: Package 'openstack-neutron-openvswitch' isn't signed with proper key
11:38:49 <kashyap> ndipanov, The compute service restarts successfully (at-least systemd reports); however, there's stack traces in compute.log -- http://paste.fedoraproject.org/66364/38909468.  I'll write a proper report
11:39:41 <ohochman> rlandy: thanks It fixed for me after I've done :  'yum --disablerepo=* --enablerepo=epel clean metadat'
11:39:45 <ndipanov> kashyap, nice
11:39:48 <ndipanov> very nice
11:40:01 <kashyap> ndipanov, I can't parse it. Is it sarcasm? :-)
11:40:09 <ndipanov> kashyap, yes
11:40:31 <kashyap> I like the blunt honesty!
11:41:00 <ohochman> jistr: I'm testing the new puppet-3.4.2-1.el6.noarch.rpm  and  will let you know where it will stuck .
11:43:23 <nlevinki> need link to download fedora20
11:44:04 <majopela> hi ohochman
11:44:11 <majopela> how did it go with foreman & rhel65 ?
11:44:19 <majopela> I'm trying the same on centos65 :)
11:44:36 <ohochman> majopela: not too good so far.
11:45:23 <mpavlase> nlevinki: check out this: http://download.eng.brq.redhat.com/pub/fedora/linux/releases/20/Fedora/x86_64/iso/
11:45:57 <majopela> ohochman, I saw this note too late:        -->   before: running foreman-server.sh / foreman-client.sh -->  'yum install -y puppet-3.3.2 puppet-server-3.3.2'
11:46:06 <majopela> good that I made an snapshot ':)
11:46:32 <ohochman> majopela: keep me up-to-date.  On my side (foreman on rhel6.5)  There are two bugs that I cannot workaround so far  :
11:46:33 <ohochman> #1047353 [RDO]: Foreman-server installation failed cause: undefined method `mode=' for #<Puppet::Settings::AutosignSetting   (workaround [1])#1048922 [RDO][Openstack-Foreman-Installer]: Deployment of controller failed (controller_common.pp:189 Wrapped exception: Name must be a String not Array).  (workaround [1])
11:47:17 <ohochman> majopela: you can try with this puppet version and let me know if it works for you.
11:48:15 <majopela> right now it's running with 3.4.2 centos default...
11:48:17 <majopela> I suppose it will fail
11:48:28 <majopela> oh no
11:48:32 <majopela> it worked :? :)
11:49:03 <ohochman> majopela: it did?  I'm doing it myself right now.
11:49:09 <ohochman> with 3.4.2
11:49:23 <majopela> it did, for the server side
11:49:26 <majopela> let's see the clients..
11:49:35 <kashyap> nlevinki, Live desktop ISO - http://download.fedoraproject.org/pub/fedora/linux/releases/20/Live/x86_64/Fedora-Live-Desktop-x86_64-20-1.iso
11:49:53 <ohochman> majopela:  good news
11:50:20 <kashyap> nlevinki, If you need a Fedora 20 (raw) cloud image -- http://download.fedoraproject.org/pub/fedora/linux/releases/20/Images/x86_64/Fedora-x86_64-20-20131211.1-sda.raw.xz
11:50:58 <nlevinki> thanks
11:51:54 <ohochman> majopela: yhep, it looks like the foreman-server-side indeed works on 6.5 with puppet  3.4.2
11:52:16 <ohochman> majopela: let's check the clients.
11:52:23 <majopela> I'm lucky today :)
11:52:37 <ohochman> majopela: which host group are you trying ?
11:52:48 <yrabl> mbourvin, http://openstack.redhat.com/Docs/Storage
11:52:49 <majopela> none yet, I didn't get to this part
11:52:54 <majopela> I started this morning :)
11:53:17 <pixelb> jistr, cool. I'll build a new openstack-foreman-installer with that patch
11:54:52 <pixelb> kashyap, Your avx issue reminds me of: https://review.openstack.org/#/c/61310/ though that's lxc related
11:55:38 <kashyap> pixelb, avx?
11:55:57 <pixelb> libvirtError: internal error: CPU feature `avx' specified more than once
11:56:29 <kashyap> Ah, right. This is the latest I see in my Compute log: http://paste.fedoraproject.org/66362/09410513
11:57:47 <kashyap> pixelb, Thanks for the review link, I'll see if I can get to root-cause of this issue.
11:59:45 <kashyap> And, yes, I still see the 'avx' trace:
11:59:45 <kashyap> 2014-01-07 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup libvirtError: internal error: CPU feature `avx' specified more than once
12:00:32 <kashyap> I bet danpb can point the fix top off his head, he knows a boat load of stuff on CPU baseline aspects in libvirt/qemu
12:01:22 <kashyap> pixelb, This looks legit too, filing a report to track it
12:01:40 <jistr> pixelb: just fyi - the patch fixes the issue, puppet manifests compile fine with Puppet 3.4.2
12:01:57 <jistr> ohochman: ^ https://github.com/redhat-openstack/astapor/pull/95
12:01:59 <pixelb> jistr, OK I notice the patched changed. I'll take the latests
12:02:28 <jistr> pixelb: yeah there should be no practical difference, but it's probably better to be in sync
12:02:51 * pixelb learned some ruby today
12:03:09 <majopela> :-) pixelb
12:04:59 <afazekas> libvirtError: internal error: CPU feature `rdtscp' specified more than once
12:06:21 <afazekas> http://www.fpaste.org/66374/90963731/
12:07:00 <ohochman> jistr: will we get new packstack-modules-puppet soon ?
12:07:13 <ohochman> jistr: rhat contain this fix https://github.com/redhat-openstack/astapor/pull/95
12:07:29 <majopela> ohochman, Error: Could not request certificate: No route to host - connect(2)
12:07:35 <majopela> hmm, but they can ping each other..
12:07:56 <ohochman> majopela: is it when you run foreman_client.sh ?
12:09:31 <ohochman> majopela: check that you can ping them by FQDN
12:09:43 <ohochman> majopela: are you using VMs  ?
12:10:27 <ohochman> majopela: I've managed to pass the foreman_client.sh stage with rhel6.5 . now trying to deploy according the host groups.
12:10:40 <eglynn> grrr, how long does PackageKit hold onto the yum lock for in a fresh f20 install?
12:10:55 <jistr> ohochman: yeah i think pixelb is doing a build with the fix included (it's not in packstack-modules-puppet, but in openstack-foreman-installer)
12:11:06 <eglynn> ... longer than a few mins it seems
12:11:17 * eglynn taps fingers impatiently ...
12:11:22 <ohochman> jistr: that means that I will have to install the foreman-server again :(
12:11:32 <majopela> yes, vms ':)
12:11:51 <jistr> ohochman: well i think that shouldn't be necessary
12:12:08 <ohochman> majopela: you'll have to check your networking . I think that jistr wrote the guide for how to make it work with VM's.
12:12:42 <ohochman> majopela: but check their FQDN first
12:13:25 <majopela> ohochman, it's my iptables
12:13:36 <ohochman> jistr: any way to get this fixed without waiting for pixelb to create the new openstack-foreman-installer rpm
12:13:37 <majopela> the foreman hosts seems quite locked down only for ssh
12:13:40 <ohochman> majopela: oh
12:13:51 <pixelb> ohochman, 2 minutes
12:14:03 <ohochman> pixelb: Ok /me waiting then..
12:14:12 <majopela> ohochman, that doesn't happen on RHEL65?
12:14:23 <majopela> I'm not sure if centos introduced any difference when building their distro
12:14:29 <jistr> ohochman: you can just apply these few lines of change https://github.com/jistr/astapor/commit/a3d554d6c2a6039f60984432bf94acadf606d7c7
12:14:53 <yfried> afazekas ping
12:14:59 <majopela> btw, lunch!! :-)
12:15:03 <majopela> see you later
12:15:18 <ohochman> majopela: :)
12:15:34 <ohochman> majopela: I set the following iptables roles on the foreman-server side :
12:15:38 <ohochman> http://pastebin.com/qQXgL7hF
12:15:40 <Dominic> majopela: no route to host indicates iptables is blocking traffic, probably port 8140/tcp on the foreman server
12:15:51 <Dominic> ^ ohochman's on it :)
12:16:13 <ohochman> *rules
12:17:10 <afazekas> yfried: pong
12:19:37 <majopela> thanks Dominic :)
12:19:45 <majopela> yes, it matches with what I found
12:21:53 <pixelb> ohochman, majopela jistr updated openstack-foreman-installer now in RDO
12:22:07 <jistr> pixelb: thanks!
12:24:21 <yfried> afazekas you are defining the repo in your yum init and then install rdo?
12:26:27 <mrunge> mbourvin, thank you for the bug report. could you please add the status of openstack-status?
12:26:54 <ohochman> pixelb: thanks
12:27:04 <mbourvin> mrunge: what do you mean by openstack-status?
12:27:05 <ohochman> pixelb: starting over...
12:27:18 <ohochman> majopela: ^^
12:27:24 <mrunge> mbourvin, issue the command openstack-status
12:28:09 <afazekas> I added rdo-ichouse to the usual packstack install script, it created a repo entry like this: http://www.fpaste.org/66379/09698413/
12:28:10 <mrunge> mbourvin, it would help, if you could check nova.api log for issues
12:29:08 <mrunge> mbourvin, neutron-openvswitch-agent:              dead does not look good for me
12:29:29 <ohochman> yfried: ping
12:29:41 <yfried> ohochman: pong
12:29:56 <ohochman> yfried: you and ofer have epel enabled on your setups/
12:29:58 <ohochman> ?
12:30:05 <yfried> ohochman: I do
12:30:10 <yfried> ohochman: don't know about ofer
12:30:17 <yfried> ohochman: did you solve the issue?
12:30:17 <afazekas> kashyap: are you using the updates-testing repo with f20 ?
12:30:30 <ohochman> and you having this problem  Execution of '/usr/bin/yum -d 0 -e 0 -y install openstack-keystone' returned 1: Error: Package: python-dogpile-cache-0.5.0-1.el6.noarch (openstack-icehouse)
12:30:37 <ohochman> yfried: ?^^
12:30:37 <mbourvin> mrunge: neutron-openvswitch-agent dead but pid file exists
12:30:45 <mbourvin> mrunge: how come it's still working
12:30:46 <mbourvin> ?
12:31:16 <mrunge> mbourvin, I can't say anything about neutron here
12:31:41 <mbourvin> mrunge: lol
12:32:03 <mrunge> mbourvin, the api logs would help, definitely
12:32:12 <yfried> yfried: I'm not. Ofer did
12:32:23 <ohochman> yfried: Ok
12:32:53 <mrunge> mbourvin, when trying to show the LaunchInstance dialog, it collects quotas first (e.g)
12:33:05 <ndipanov> pixelb, could you confirm that novnc is broken?
12:33:13 <ndipanov> pixelb, just go to console in horizon
12:33:37 <pixelb> ndipanov, not in a position to do that at present
12:33:48 <ndipanov> pixelb, k
12:33:51 <ndipanov> kashyap, ^
12:38:22 <pixelb> mbourvin, the neutron-openvswitch-agent issue is already logged. You need to `yum install python-psutil` before starting that service
12:39:46 <pixelb> That is a temp workaround of course until the bug is fixed. kashyap have to logged that bug and workaround on RDO wiki?
12:39:55 <afazekas> http://www.fpaste.org/66388/90983821/
12:42:38 <afazekas> I see similar errors in the n-cpu logs
12:43:13 <pixelb> afazekas, kashyap had a very similar issue, but with "avx" rather than "rdtscp". Something dodgy is going on with these cpu flags
12:44:28 <afazekas> As I remember the baseLine related code can be commented out
12:49:04 <Dafna> tshefi giulivo i'm back, was it the same bug from ystday?
12:49:34 <giulivo> Dafna, nope, icehouse is setting that appropriately
12:50:08 <Dafna> giulivo: what was it? iptables? :)
12:50:23 <tshefi> Dafna, giulivo ,   but the name of qpid server was blank
12:50:29 <giulivo> no the parameter was blanked but on my deployment, using same version of packstack, it was set
12:50:43 <giulivo> so let us know if it works for you
12:51:14 <giulivo> and see if we can find the root cause, for now I moved on a different issue with the emc backend driver
12:51:16 <Dafna> giulivo: I am configuring gluster as my glance backend but will try to create an image right after...
12:51:49 <giulivo> Dafna, yeah just check if in glance-api.conf you have qpid_hostname set
12:52:26 <Dafna> tshefi can you ping flavio? fpercoco is his nick and he should be on rhev-dev
12:53:54 <giulivo> Dafna, sec, I'm not sure how to reproduce this first
12:54:09 <giulivo> that's why I'm investigating it a bit
12:54:25 <giulivo> it is eventually an issue with packstack not glance
12:55:05 <Dafna> giulivo: lets find what the issue is first and than worry about reproducing it...
12:55:23 <giulivo> yeah so how is qpid_hostname for your glance-api.conf ?
13:01:21 <afazekas> kashyap: http://www.fpaste.org/66395/13890996/
13:02:31 <Dafna> yrabl: ping
13:07:36 <Dafna> giulivo: did we end up opening the bug on /etc/cinder/shares.conf being created as root:root?
13:09:27 <giulivo> no I haven't
13:09:30 <giulivo> Dafna, ^^
13:10:13 <Dafna> giulivo: cool. it append to me and yrabl - yogev, open ok?
13:10:40 <yrabl> Dafna, giulivo cook
13:10:42 <yrabl> cool
13:12:40 <Dafna> giulivo: cinder.service ProgrammingError: (ProgrammingError) (1146, "Table 'cinder.services' doesn't exist")
13:12:51 <Dafna> giulivo: that is after I fixed the permission issue...
13:13:14 <giulivo> wow :)
13:14:06 <giulivo> Dafna, but it doesn't seem to be related to gluster, you should have that table
13:14:17 <giulivo> or any driver won't work
13:14:18 <Dafna> giulivo: this is odd...
13:14:36 <weshay> panda, rlandy how are the installs going
13:14:43 <Dafna> sql_connection=mysql://cinder:a597d1d2bb9245da@10.35.xx.xx/cinder
13:14:49 <ohochman> pixelb: do you know why I'm getting this "GPG key retrieval failed: [Errno 14] Could not open/read file:///etc/pki/rpm-gpg/RPM-GPG-KEY-puppetlabs"
13:14:59 <Dafna> giulivo: it should connect to the correct machine...
13:15:21 <pixelb> ohochman, that key should be in the rdo-release.rpm Have you that installed?
13:15:30 <giulivo> Dafna, yeah but apparently it doesn't find the "services" table in the "cinder" db
13:15:35 <ohochman> pixelb: right.
13:15:38 <giulivo> so it is connecting, but the db seems broken
13:15:41 <ohochman> pixelb: thanks
13:15:53 <Dafna> giulivo: yeah, I'm logging in to the db now to check
13:16:50 <rlandy> weshay: mixed results
13:17:11 <weshay> partly cloudy
13:17:11 <Dafna> giulivo: yaykes...
13:17:15 <Dafna> giulivo: mysql> use cinder;
13:17:15 <Dafna> Database changed
13:17:15 <Dafna> mysql> show tables;
13:17:15 <Dafna> Empty set (0.00 sec)
13:17:34 <giulivo> Dafna, I think something went wrong during install though
13:17:37 <rlandy> yeah ... install did a lot better
13:17:48 <rlandy> no workarounds required although process took a lot longer
13:18:02 <Dafna> giulivo: I think so too...
13:18:23 <Dafna> giulivo: but all was reported as ok
13:18:32 <giulivo> are you trying this on rhel6.5?
13:18:41 <Dafna> yrabl: can you please check your db?
13:18:48 <Dafna> giulivo: of course
13:19:09 <Dafna> giulivo: but I think I know what happened... and it's a bug...
13:19:39 <rlandy> weshay: hit this error when running an instance ... can see from the comments that packstack does not setup all the config params that may be needed
13:19:44 <rlandy> https://bugzilla.redhat.com/show_bug.cgi?id=1020954
13:20:08 <yrabl> Dafna, mine is ok
13:20:10 <Dafna> giulivo: however, I am not sure why it only happened to cinder...
13:20:16 <Dafna> yrabl: thanks
13:20:28 <Dafna> yrabl: you are using the ice-house pacakges right?
13:20:49 <yrabl> Dafna, yes. I was briefed :)
13:20:50 <giulivo> Dafna, I have it installed too on rhel and the db is okay
13:21:07 <Dafna> yrabl: :)
13:21:25 <yrabl> Dafna, and the db is on a different server :)
13:21:49 <panda> weshay: RHEL 6.5 distributed install OK, working on ML2 plugins: installs fine (workarounded psutils bug) now creating some network, subnet ...
13:21:55 <Dafna> yrabl: thanks.
13:22:12 <weshay> bah.. you have to hate it when those bugs are just closed..  should be reassigned if the devel thinks its a packstack issue
13:22:15 <Dafna> giulivo: I think its one of those things that can only happen to me...
13:22:43 <Dafna> giulivo: still a bug...
13:22:48 <weshay> panda, cool.. /me checks on the multinode job test results
13:23:06 <rlandy> guess it's a design choice ... either way, that was my "get yourself acquainted' install .. moving on the to F20 one I actually signed up for noe
13:23:07 <rlandy> now
13:24:01 <Dafna> giulivo: I just re-installed my controller and ran packstack... my guess is that packstack assumed that tables exists... I am just not sure why it did not do the same for glance
13:24:24 <Dafna> giulivo: actually... it did...
13:25:33 <giulivo> rlandy, hi :) I got an issue with mysqld on f20 which I'm uncertain about; it is not the problem mentioned in the workaround, do you plan to install via packstack or foreman?
13:26:22 <panda> weshay: how do I publish tempest results for ML2 ?
13:27:26 <weshay> panda, I think fpaste may be an option
13:27:42 <rlandy> giulivo: hi ... long time .. how are you? I am going to follow the Using_GRE_Tenant_Networks - so packstack
13:30:35 <yrabl> Dafna, did you tried to create a volume from an image?
13:31:47 <Dafna> yrabl: no. I did not have a db created :)
13:32:13 <jruzicka> RHEL 6.5 works for me without workarounds.
13:33:55 <ohochman> majopela: jistr pixelb : good news - foreman in rdo  (!)  Using  the:   latest & greatest: -  openstack-foreman-installer noarch 1.0.1-2.el6  +  puppet-3.4.2-1.el6.noarch.rpm , the first scenario: "2 Node install with Nova Networking" -  works good  (tested with : rhel6.5).
13:34:10 <pixelb> ohochman, excellent
13:34:22 <jistr> ohochman: cool :)
13:34:23 <vladan> fedora 20 works for me with the mysql to mariadb packstack workaround
13:34:36 <ohochman> :-D
13:34:47 <majopela> ohochman, coool! :D
13:35:10 <majopela> I had a meeting, and in just back into it :)
13:35:17 <ohochman> pixelb: jistr majopela: I'll try the neutron host groups now.
13:40:45 <flaper87> any of you guys seeing this? http://paste.openstack.org/show/60634/
13:41:11 <vladan> flaper87: yes, you need to edit a file
13:41:13 <vladan> sec
13:41:28 <vladan> flaper87: https://bugzilla.redhat.com/show_bug.cgi?id=1012786#c4
13:41:35 <flaper87> I didn't find anything in the workarounds page
13:41:37 <flaper87> lemme add that
13:41:51 <jruzicka> flaper87, thanks for adding that
13:42:04 <jruzicka> Workarounds page is convenient
13:42:34 <flaper87> jruzicka: if you give me $1000 I promisse I wont make something up
13:42:43 * flaper87 giggles
13:43:27 <jruzicka> flaper87, hmm, I can live with that...
13:49:08 <flaper87> http://openstack.redhat.com/Workarounds#Could_not_enable_mysqld
13:49:11 <flaper87> there you go
13:49:50 <ndipanov> pixelb, looks like we have a wrong version of websockify in the repo
13:50:07 <pixelb> ndipanov, checking
13:50:17 <ndipanov> 4.1
13:50:20 <ndipanov> and we need 5.1
13:50:32 <ndipanov> pixelb, checking fedora now
13:50:54 <eglynn> kwhitney: hi
13:51:22 <pixelb> ndipanov, I've 0.5.1 in el6. So f20 needs an update ?
13:51:43 <ndipanov> likely pixelb will do it now
13:51:49 <eglynn> kwhitney: if you're testing ceilometer on RDO/icehouse, note the following issue https://bugzilla.redhat.com/1049369
13:52:01 <ndipanov> will you handle the repo updates (how does it go now since jruzicka was working on it)
13:52:19 <kwhitney> eglynn: OK will check
13:52:33 <eglynn> kwhitney: ... and the corresponding workaround: http://openstack.redhat.com/Workarounds#ceilometer:_notifications_from_openstack_services_not_processed
13:52:45 <ndipanov> pixelb, ^ ?
13:53:00 <pixelb> ndipanov, will do
13:53:10 <ndipanov> pixelb, thanks
13:54:37 <pixelb> ndipanov, Also not one should be using updates-testing really, which does contain the right version
13:55:18 <ohochman> pixelb: the neuron controller deployment ended with the client in  "E" status
13:55:33 <ohochman> but I don;t see any errors in the puppet agent outputs
13:55:33 <pixelb> E for Excellent?
13:55:40 <ohochman> Yes :D
13:57:24 <Egyptian[Laptop]> morning
13:57:26 <ohochman> pixelb: I had this tinny error : http://pastebin.com/8ErMErTf  (from /var/log/messages
13:57:47 <Egyptian[Laptop]> how come the rdo test day isnt on planet fedora?
13:58:03 <ohochman> pixelb: not sure that's the reason for the "E"
13:59:37 <pixelb> Egyptian[Laptop], fair point. It has diverged a bit from Fedora test days, but it would have been useful to mention it there
14:01:56 <kwhitney> eglynn: I don't see the agent-notifier in "chkconfig" list of auto-start services.  SHouldn't it be there as well?
14:03:42 <eglynn> kwhitney: yes, as per the usual pattern for required services, i.e. ... service foobar start ; chkconfig foobar on
14:04:03 <eglynn> kwhitney: (so both steps are missing for the new notification agent)
14:04:32 <eglynn> kwhitney: in the BZ I didn't explicitly mention chkconfig (took it as implicit)
14:04:47 <kwhitney> OK.  I wondered because that was not mentioned in the work-around or the bug.  Just needing to start a service is different than not being listed to auto start
14:04:49 <eglynn> kwhitney: ... so feel free to append a comment to the bug report
14:05:12 <kwhitney> eglynn: Thx.  Will do.
14:05:26 <rlandy> giulivo: ok ... I have a mysql error on F20
14:05:48 <rlandy> pasting error for you to review
14:05:54 <giulivo> rlandy, so this is probably the one mentioned in the workaround
14:06:07 <giulivo> http://openstack.redhat.com/Workarounds_2014_01
14:06:11 <kashyap> ndipanov, pixelb: /me went out for a walk (weather was too good), now reading the scroll
14:06:37 <giulivo> rlandy, if that is the case, try to go trough the suggested steps and launch again packstack using the *same* answer file
14:06:38 <eglynn> kwhitney: thank you sir!
14:06:47 <ndipanov> kashyap, I fixed my issue so dissregard me
14:07:42 <kashyap> ndipanov, Ok. And, I see that the CPU flags issue is legit. . .
14:07:50 <ohochman> pixelb: what are you saying about the mongodb error (during neutron controller deployment) ?
14:08:00 <rlandy> giulivo: not the one in the workaround but the one in the BZ .. yeah doing that
14:08:14 <ohochman> pixelb: http://pastebin.com/8ErMErTf
14:08:26 <pixelb> ohochman, that looks like an epel access issue? can you `yum install v8` on that node manually?
14:08:30 <majopela> ohochman, pixelb
14:08:34 <majopela> it fails for me on the clients
14:08:39 <ndipanov> kashyap, bug?
14:08:39 <majopela> with the string / array bug
14:08:48 <ohochman> majopela: with nova-network ?
14:08:48 <kashyap> ndipanov, On its way
14:08:52 <majopela> neutron
14:09:03 <ndipanov> kashyap, thanks!!
14:09:45 <ohochman> majopela: can you check the puppet-modules version on your foreman-server ?
14:10:04 <majopela> ohochman, how do I do that?
14:10:19 <ohochman> majopela: should be :packstack-modules-puppet-2013.2.1-0.27.dev936.el6.noarch
14:10:29 <ohochman> rpm -qa | grep packstack-modules-puppet
14:10:39 <pixelb> rpm -q packstack-modules-puppet
14:10:40 <majopela> oh, ok ok, the rpms
14:10:54 <majopela> I was thinking on querying puppet sorry, I need more coffee :)
14:11:05 <majopela> packstack-modules-puppet-2013.2.1-0.27.dev936.el6.noarch
14:11:58 <ohochman> majopela: have you started the foreman-server installation with this version  ?
14:11:58 <pixelb> majopela, note also that openstack-foreman-installer needs to be 1.0.1-2
14:12:06 <ohochman> and that ^^
14:12:10 <majopela> hmm rechecking
14:12:30 <majopela> openstack-foreman-installer-1.0.1-1.el6.noarch
14:12:31 <ohochman> foreman-installer-1.3.1-1.el6.noarch
14:12:38 <majopela> I'm outdated!
14:12:39 <majopela> :-)
14:12:42 <ohochman> |'m using  ^^
14:12:44 <ohochman> foreman-installer-1.3.1-1.el6.noarch
14:13:03 <majopela> the foreman installer seems fine
14:13:24 <pixelb> ohochman, foreman-installer hasn't been updated. majopela you need to `yum install openstack-foreman-installer` to get the latest
14:13:25 <ohochman> majopela: seems - but it's not ! (in this version)
14:13:43 <ndipanov> btw is it a feature or a bug that I don't get the demo rc file created in my home dir after packstack?
14:13:48 <ndipanov> mmagr, ^
14:13:59 <majopela> Package openstack-foreman-installer-1.0.1-1.el6.noarch already installed and latest version
14:14:09 <majopela> weird... caching?
14:14:24 <pixelb> majopela, yum --disablerepo=* --enablerepo=openstack-icehouse clean metadata
14:14:44 <ohochman> majopela: as pixelb said --> you need to have :  openstack-foreman-installer-1.0.1-2.el6.noarch
14:14:48 <pixelb> majopela, actually different repo name, so
14:15:08 <majopela> got it updated :-)
14:15:17 <ohochman> it won't do it ^
14:15:23 <ohochman> you'll have to reinstall
14:15:50 <ohochman> pixelb: regarding the neutron ..
14:15:59 <morazi> ohochman, nice work on the first foreman scenario!  iirc there might be one outstanding packstack-modules issues that may require a manual tweak.  lemme see if that pastebin is what I"m thinking of...
14:16:05 <majopela> hmm, I need to reinstall? :)
14:16:17 <majopela> ok, back to previous snapshot ... sniff :'( :)
14:16:28 <ohochman> majopela: indeed :D
14:17:30 <ohochman> morazi: there's a problem I cannot nail with the neutron hostgroup,  It fails to install mongodb ? -  http://pastebin.com/8ErMErTf
14:18:02 <ohochman> morazi: not sure yet what's the issue. but the neutron controller is in "E" after deployment.
14:18:14 <mmagr> ndipanov, it depends ... it is creating rc files in homedir only in all-in-one installations
14:18:46 <ohochman> pixelb: v8 can be installed manually.
14:18:48 <ndipanov> mmagr, yeah - I did a allinone that failed but reused the answer file
14:18:53 <ndipanov> mmagr, so that's expected then
14:19:17 <pixelb> ohochman, I don't know what puppet would be doing differently TBH
14:19:22 <ndipanov> mmagr, ?
14:19:22 <kashyap> ndipanov, pixelb: There we go - https://bugzilla.redhat.com/show_bug.cgi?id=1049391
14:19:37 <kashyap> ndipanov, Logs coming in attachment
14:19:57 <pixelb> ohochman, maybe transient net error?
14:19:59 <ohochman> pixelb: maybe it just got temporary hang -  No more mirrors to try.
14:20:05 <ndipanov> kashyap, thanks
14:20:11 <ohochman> pixelb: yheppp
14:20:40 <ohochman> pixelb: I wonder if running the puppet agent again will fix it.
14:23:26 <morazi> ohochman, probably worth giving it a shot to re-run and see.
14:24:38 <morazi> ohochman, out of curiosity, what did you list what repos you have enabled?
14:25:14 <ohochman> morazi: epel.repo  foreman.repo  puppetlabs.repo  rdo-release.repo  rhel6_update.repo  rhel-optional.repo  rhel-server.repo  rhel-source.repo  rhel-updates.repo
14:25:18 <rlandy> ukalifon: hi .. how far did you get with the 'allinone' rhel6.5 test? I hit some errors on running on instance
14:25:54 <weshay> aio rhel6.5 should work w/ I think three workarounds
14:26:17 <pixelb> flaper87, You edited the "havana" workarounds page I think. Also note there is an existing workaround for the mysqld issue on the test day workarounds page: http://openstack.redhat.com/Workarounds_2014_01
14:26:36 <ukalifon> rlandy: I never succeeded in getting an instance to run
14:26:40 <weshay> Failed to parse /etc/nova/nova.conf (RHEL)
14:26:54 <flaper87> pixelb: mmh, sorry, too many tabs! Let me move that
14:27:13 <pixelb> flaper87, s/move/remove/ possibly?
14:27:24 <flaper87> pixelb: move havana -> icehouse
14:27:53 <pixelb> flaper87, OK if you think the existing icehouse workaround isn't appropriate
14:28:17 <pixelb> weshay, Those three bugs associated with "Failed to parse /etc/nova/nova.conf" should be fixed I think
14:28:32 <weshay> ok.. will remove the wrkaround
14:29:45 <Dafna> yrabl: ping
14:29:51 <flaper87> pixelb: I kept the existing workaround but added one more step
14:29:54 <vladan> flaper87: pixelb I'd leave this workaround because that's the way it should be done in packstack, moving service files seems much dirtier
14:29:55 <morazi> ohochman, I'd be curious to see if it is just a blip and/or what happens if you try to manually yum install mongodb-server on that box
14:29:56 <ohochman> morazi: I"ve added the repo list to : https://etherpad.openstack.org/p/rdo_test_day_jan_2014
14:30:01 <pixelb> flaper87, great thanks
14:30:26 <vladan> imho :)
14:30:32 <flaper87> vladan: yeah, I kept the old steps because there's an "If you hit this when running packstack..." section
14:30:44 <kashyap> pixelb, Added the psutil note to the Workarounds page.
14:30:45 <pixelb> vladan, Moving service files is to avoid a systemd bug. IMHO I think packstack should be able to use mysqld directly
14:30:54 <vladan> aha k
14:32:27 <vladan> pixelb: but if mysqld doesn't exist as a service (which is the case now) it shouldn't try to start it at all
14:32:41 <vladan> pixelb: which systemd bug are you referring to?
14:33:18 <ohochman> morazi: jistr pixelb : I ran puppet agent second time on the neutron controller (and it failed on keystone token-get ? http://pastebin.com/6FBdqQt1
14:33:52 <ohochman> maybe I'll try neutron-controller again on clean machine
14:34:04 <weshay> FYI.. execute tempest on icehouse:
14:34:06 <weshay> http://openstack.redhat.com/Testing_IceHouse_using_Tempest#configure_and_run_nosetest_.28_RHEL.2FCentOS_.29
14:34:20 <pixelb> vladan, mariadb tried to provide compat so that one could still support using the mysqld service name. This was done with symlinks. systemd changed recently to have issues with that. The service copy is just to convert a symlink to a real copy. There is no systemd bug yet
14:34:40 <Dafna> giulivo: https://bugzilla.redhat.com/show_bug.cgi?id=1049382 can you please copy the error in the bug comments?
14:34:46 <rlandy> giulivo: second packstack run with the same answer file completed w/o error
14:35:10 <larsks> pixelb: DO you have any links with more information about the systemd/symlink thing?
14:35:52 <morazi> ohochman, that seems reasonable.  I suspect you got wedged halfway through due to the package blip and it left the data store behind keystone in an inconsistent state (or simply foreman didn't know how to resume against an existing keystone db)
14:35:57 <vladan> pixelb: I see, thanks for calrifying
14:36:39 <pixelb> larsks, All details I have are in: https://bugzilla.redhat.com/981116 I'll log a systemd bug at some stage
14:36:54 <ohochman> morazi: ok , I'll try it again with clean node
14:37:12 <giulivo> rlandy, great, thanks
14:37:45 <larsks> pixelb:  thanks!
14:37:47 <giulivo> Dafna, yep but that is failing because of an issue with the SMI-S
14:37:56 <giulivo> I'm checking that too
14:38:09 <ohochman> morazi: BTW - I do have many AVC here.
14:38:35 <Dafna> giulivo: not saying it's not, it just makes it easy to search BZ's later when the trace is in the bug
14:38:43 <giulivo> Dafna, I will
14:38:53 <Dafna> giulivo: thanks
14:40:46 <morazi> ohochman, same as:  https://bugzilla.redhat.com/show_bug.cgi?id=1043887 ?
14:40:47 <afazekas> Kashyap: did you tried the http://www.fpaste.org/66395/13890996/ ?
14:41:24 <kashyap> afazekas, No, I was investigating the avx CPU flags bug - https://bugzilla.redhat.com/show_bug.cgi?id=1049391
14:41:37 <kashyap> afazekas, What is this patch for?
14:42:02 <afazekas> kashyap: probbaly a workaround for that bug
14:42:37 <kashyap> "probably" :-)
14:43:39 <pixelb> kashyap, eglynn I've moved your workarounds from the havana to icehouse page in case you're wondering where they went: http://openstack.redhat.com/Workarounds_2014_01
14:44:09 <afazekas> kashyap: my novanet instance had network issue after that, the neutron was already ran out from the ports, and I had to be away, now I am resintaling and testing the workaround
14:44:12 <kashyap> pixelb, Sure, thank you. I'd have done that myself had I noticed this page
14:44:29 <kashyap> afazekas, You using devstack?
14:45:36 <pixelb> afazekas, kashyap I'll add that to the workarounds page
14:45:40 <afazekas> kashyap: https://review.openstack.org/#/c/63647/
14:46:23 <afazekas> Now with devstack you would need to use the FORCE option and installing some extra packages before starting
14:47:34 <afazekas> BTW it was the n-net issue: http://www.fpaste.org/66420/10603913/
14:48:08 <ohochman> morazi: it's :  avc:  denied  { read } for  pid=26034 comm="ip" path="/proc/25941/status" dev=proc ino=42683 scontext=unconfined_u:system_r:ifconfig_t:s0 tcontext=unconfined_u:system_r:initrc_t:s0 tclass=file
14:48:22 <afazekas> pixelb: ok
14:48:35 <ohochman> morazi: I don't see it in this bug : https://bugzilla.redhat.com/show_bug.cgi?id=1043887
14:48:36 <kashyap> afazekas, Noted; pixelb - thanks
14:50:31 <ohochman> morazi: I think It's becasue the AVCs in this bug are from the foreman-server installation stage and this AVC appears on the foreman-client while deploying neutron-controller.
14:51:07 <majopela> ohochman, pixelb , (neutron) controller installed :-)
14:51:41 <pixelb> majopela, woot!
14:52:23 <nmagnezi> kashyap, the workaround for ovs agent (the one with gpgcheck=0) did not work for me.
14:52:31 <nmagnezi> kashyap, any other ideas? :-)
14:54:35 <kashyap> nmagnezi, Are you referring to this? - https://bugzilla.redhat.com/show_bug.cgi?id=1049235
14:55:39 <morazi> majopela, nice one!
14:55:42 <nmagnezi> kashyap, actually no, I was reffering to another workaround. but i'll try this one now
14:55:54 <ohochman> majopela: does it's status "A"
14:55:55 <ohochman> ?
14:56:13 <ohochman> on the foreman-server gui
14:56:16 <kashyap> nmagnezi, If it's related to ImportError: No module named psutil ; the above note I mentioned should work
14:56:44 <majopela> ohochman, yes :)
14:57:02 <ohochman> majopela: Ok :D
14:57:20 <nmagnezi> kashyap, it does! :)
14:57:27 <nmagnezi> kashyap, and it worked
14:58:07 <ohochman> morazi: majopela  : worked for me too.
14:58:48 <ayoung> Just logging in now.  Any Keystone issues cropped up thus far?
14:59:02 * ayoung asssumes test day chatter is happening here?
14:59:10 <kashyap> nmagnezi, Cool.
14:59:44 <majopela> hmm
14:59:48 <majopela> on compute node I found this:
15:00:02 <ohochman> majopela: morazi : now we are having the same neutron havana bugs -  all neutron services are down + /etc/init.d/neutron-openvswitch-agent cannot be started :D
15:00:12 <ohochman> but that's "NORMAL"
15:00:14 <majopela> Error: failure: repodata/44b20c52ea5f27ef34b267d98b79cafd52ca9533340d8fb94377506a380f3d3d-filelists.sqlite.bz2 from openstack-icehouse: [Errno 256] No more mirrors to try.
15:00:14 <majopela> You could try using --skip-broken to work around the problem
15:00:14 <majopela> You could try running: rpm -Va --nofiles --nodigest
15:00:51 <majopela> yes, ohochman , what you say is because of python-psutil, right?
15:01:10 <ohochman> majopela: how do you know that
15:01:11 <ohochman> ?
15:01:31 <majopela> I think I read it in some BZ
15:02:02 <ohochman> well, I don't think so.  when I'm tying to : /etc/init.d/neutron-openvswitch-agent start
15:02:17 <majopela> https://bugzilla.redhat.com/show_bug.cgi?id=1049235
15:02:21 <majopela> is not that one?
15:02:22 <ohochman> Package 'openstack-neutron-openvswitch' isn't signed with proper key
15:02:38 <jruzicka> huh
15:02:38 <majopela> woops :)
15:03:01 <ohochman> majopela: from F20 ?
15:03:08 <majopela> no, centos65
15:03:24 <morazi> ayoung, there was something that hasn't been reported.  stack trace was like:  http://pastebin.com/6FBdqQt1 (from ohochman about 30 minutes ago)
15:03:36 <ayoung> morazi, OK, looking
15:03:57 <ohochman> ayoung: if you need to check on the machine let me know.
15:04:19 <morazi> ayoung, but it is an odd-ball -- the foreman install failed for other reasons and on the second run of puppet we run into an auth issue.  My presumption is either the keystone datastore was partially set up or that foreman doesn't have enough info to resume properly
15:04:22 <ayoung> ohochman, is there a keystone log?
15:04:32 <ohochman> kashyap: is there a workaround for this ? https://bugzilla.redhat.com/show_bug.cgi?id=1049235
15:04:44 <ohochman> kashyap: happened  to me on rhel6.5
15:05:01 <ayoung> morazi, is Keystone running on that machine?
15:05:24 <kashyap> morazi, ayoung FYI -This channel is logged w/ meetbot for today & tomorrow; Later you can retroactively grep for keystone or other keywords from the meeting file.
15:05:31 <kashyap> ohochman, Looking
15:05:52 <kashyap> ohochman, Yes
15:06:02 <kashyap> ohochman, $ yum install python-psutil -y
15:06:14 <ohochman> ayoung: most of it is : WARNING keystone.common.utils [-] Deprecated: v2 API is deprecated as of Icehouse in favor of v3 API and may be removed in K.
15:06:27 <ohochman> ayoung: no errors
15:06:37 <ohochman> kashyap: thanks..
15:06:38 <ayoung> ohochman, yeah, give me access
15:07:20 <mbourvin1> I get this error while trying to create a disk that is bigger then 20G: No valid host was found
15:07:29 <mbourvin1> did anyone see this error before
15:08:34 <kashyap> mbourvin1, Obvious check:  probably none of your compute hosts have enough space?
15:08:50 <kashyap> mbourvin1, You can check your Nova scheduler & api logs for more clues.
15:08:51 <majopela> ok the  yum --disablerepo=* --enablerepo=openstack-icehouse clean metadata  did the trick to get the openstack-nova-compute installing
15:14:27 <majopela> ohochman, pixelb , compute is up!!, going for the network server
15:14:35 <shardy> Hey all, is it necessary to make the em1 interface not controlled by network manager when enabling neutron for RDO installs on F20?
15:15:07 <shardy> I have one box w/nova-network and network-manager works fine, but the box with neutron drops the network and it never comes back
15:15:24 <majopela> you should disable network manager, right
15:16:11 <shardy> majopela: Ok, thanks - we should probably mention that in the quickstart then, now that neutron is used by default
15:19:17 <afazekas> strange I do not have any net namspace in neutron install..
15:20:34 <ohochman> Dominic: morazi: I think I just encountered a problem :  I have several hosts associated with neutron-controller host group.  When I changed the General Host Group parameters It changed the parameters of all the hosts that were already associated with that host group.
15:20:41 <afazekas> tenant_network_type=local and I have 2 network and router..
15:21:05 <Dominic> ohochman: if you change a host group parameter then it will affect all hosts with that group, yes
15:21:19 <ohochman> Dominic: moraziIt ran over the values I had for each specific host.
15:21:21 <Dominic> you can override a host group parameter when you edit an individual host though
15:21:39 <morazi> ayoung, sorry no visiblity on that machine at all.  Just saw you pop on and was thinking when I originally saw the stack trace I'd really love someone who knows keystone to have a peak and weigh in on it.  ;-)
15:21:56 <afazekas> I do not have missing net ns  issue with RHEL 6.5
15:22:12 <ohochman> Dominic: I thought It's a default value when adding host with that host group - but not change already exist hosts.
15:22:29 <ayoung> morazi, I'm getting the moring barrage of questions right now, but I'm looking into it.  Keystone seems to be healthy.  Need a keystonerc file
15:23:58 <ohochman> Dominic: I think it's too dangerous to allow such operation.
15:24:17 <Dominic> ohochman: you've lost me tbh, I don't understand the problem
15:26:23 <kashyap> afazekas, Hey, were you able to test that patch for the CPU baseline problem/
15:27:04 <ohochman> Dominic: If you have in foreman 100 computes-nodes managed by 2 controllers and the user change the controller IP  on the compute host group parameter it will change the parameters for 100 up running hosts.
15:27:11 <ohochman> I just think it's too dangerous to allow such operation.
15:27:57 <morazi> Dominic, maybe we need to make sure it is clear the precedence/order of operations.  how does the param on a host group get overriden by a value on an individual host.
15:27:57 <Dominic> ohochman: that's a feature.. if you have junior admins, you can create users with restricted permissions so they can't change these settings
15:27:58 <majopela> pbrady, ohochman , I have the networking node!! woah :)
15:28:54 <Dominic> morazi: sure, docs or something else?  They're shown as inherited parameters when creating/editing hosts with an override there.
15:29:57 <morazi> Dominic, I think I was speaking more to my own ignorance of the details more than anything else.  ;-)  I bet it is already documented up somewhere in foreman land
15:30:10 <Dominic> maybe :)
15:30:17 <afazekas> kashyap: Now I am failing on the f20 networking tests
15:30:51 <morazi> ohochman, yea, the flip is that you would right bespoke scripts to changes things on 100 hosts (or worse yet, assign some poor sucker to ssh into each host and make the change -- and sure fat-finger that value at least once)
15:31:00 <ayoung> ohochman, morazi I need a keystonerc for that machine, or at least admin/password
15:31:47 <ohochman> morazi: Dominic not in case those parameters are being the default for new hosts .
15:32:14 <afazekas> kashyap: http://www.fpaste.org/66439/38910871/
15:32:28 <ohochman> Dominic: morazi : and btw , I think that in foreman (the full version) you can click select and edit few hosts at once..
15:32:50 <morazi> ohochman, ^^ ayoung request was to dig into that keystone problem you were seeing -- is that host still active or did you recycle that to run a different test?
15:32:58 <ayoung> morazi, I'm on ity
15:33:10 <morazi> ayoung, cool
15:33:14 <kashyap> afazekas, As a side note - there's a similar CPU baseline bug, for LXC, pointed by pixelb here -  https://review.openstack.org/#/c/61310/
15:33:51 <Dafna> giulivo: did you also open a launchpad for the bug? :)
15:33:57 <afazekas> and features != -1 was libvirt bug
15:34:18 <kashyap> afazekas, You have a URL?
15:34:47 <morazi> ohochman, yea, so it is potentially an order of operations thing perhaps?  I suspect how things would have to work under the covers for foreman is that it would apply changes to the host group, and then re-override values for each individual host.
15:34:58 <giulivo> Dafna, nope let me update the bug so you'll know why
15:35:14 <giulivo> I'm looking into the SMI-S , this is basically blocking the tests
15:35:29 <kashyap> afazekas, That tempest n/w test failure - is that a single-node test? I presume so
15:35:35 <Dafna> giulivo: ok. Glad to hear ;)
15:35:42 <afazekas> http://www.redhat.com/archives/libvir-list/2013-November/msg00967.html , https://bugzilla.redhat.com/show_bug.cgi?id=1033039
15:35:48 <yfried> kashyap: did you try this workaround?
15:35:59 <kashyap> "this"?
15:36:03 <morazi> ohochman, so the questions/things we should confirm (I think) are -- does the over-ride get re-applied to an individual host?  What is the contract around atomicity there?  Dominic may know the answers there and can explain if my wild conjecture above ^^ has any basis in reality
15:36:14 <yfried> kashyap: https://bugzilla.redhat.com/show_bug.cgi?id=1049235
15:36:15 <ayoung> morazi, ohochman the problem might be the endpoint definition.  They all have urls like this: http://10.35.161.188:5000/v2.0  but your command was attempting to contact them using 127.0.0.1
15:36:26 <yfried> kashyap: not a fast copy/past -er
15:36:43 <ohochman> morazi: I would guess that the host-groups  parameters are the default for new hosts but changes to the host groups won't effect all hosts already within that host group (which the user might already manually changed parameters for each one of them).
15:37:05 <kashyap> yfried, Yes - that's what fixed that prob. nmagnezi confirmed it too. I noted the root-cause of it in Comment#1
15:37:23 <Dominic> morazi: sorry, I don't really understand
15:37:28 <ohochman> ayoung: the machine IP  10.35.161.188  and the 127.0.0.1 is local host ?
15:37:38 <Dominic> ohochman: changes to a host group will affect all hosts with that group assigned, next time they run Puppet
15:37:44 <nmagnezi> yfried, kashyap , yup - the ovs agent is up and running
15:37:52 <ohochman> Dominic: very bad IMHO
15:37:57 <ayoung> ohochman, right
15:38:01 <afazekas> http://www.fpaste.org/66440/13891090/ n-net f20
15:38:05 <Dominic> ohochman, morazi: if you override a value on a host then it will pick up that value instead next time Puppet checks in
15:38:14 <ohochman> ayoung: so what's the problem ??
15:38:22 <Dominic> ohochman: it's a feature, you're meant to be able to change configuration for all hosts in a group
15:38:22 <morazi> ohochman, yea, I can also see it being implemented in that way -- however that would likely orphan things if you say added a new parameter to the host group.  All the old individual hosts would would not contain the new param
15:38:36 <ayoung> ohochman, not sure.  I need a keystonerc file to move ahead
15:38:44 <yfried> nmagnezi: I had another issue - after the fix secgroup db was damaged
15:38:53 <yfried> nmagnezi: or secgroup api
15:39:32 <Dominic> morazi, ohochman: we've loosely talked about versioning these things so you can do more gradual rollouts, but that's not how it works today
15:39:33 <yfried> yfried: could be because i ran /etc/init.d/neutron-ovs-cleanup accidentally
15:40:07 <nmagnezi> yfried, i did not encounter this. but i do not focus on sec-groups as much as you do.
15:40:20 * afazekas the command is working when manually running: http://www.fpaste.org/66443/10920413/
15:41:32 <morazi> Dominic, struggling with how to rephrase.
15:43:38 <morazi> Dominic, I think maybe the question is -- if you have a host group with some members and a particular host overrides one of the params and runs puppet we would get the overridden value landing on the first puppet run.  If we subsquently update the host group, does the overridden param get erased for all intents and purposes?
15:44:03 <Dominic> morazi: ah ok, no, the host's overridden value will always take precedence over the value in the host group
15:44:05 <giulivo> Dafna, so it looks like cinder is implementing a broken workflow
15:44:18 <Dafna> giulivo: how do you mean?
15:44:36 <morazi> ohochman, I think fundamentally that's what you are asking and observing that you are losing an overridden value somewhere, correct?
15:44:38 <giulivo> in the way it performs the operation on the SMI-S
15:45:05 <giulivo> this should definitely be open upstream and most probably needs an eye from EMC people
15:45:52 <ohochman> morazi: think of a scenario when user is changing the admin_password parameter in the compute hostgroup , when he have 100 compute-nodes running and register on different controller -  if puppet will be trigger his cloud might collapse.
15:46:21 <ohochman> morazi: I woudn't want to manage my cloud with foreman allowing such action .
15:46:40 <ayoung> morazi, ohochman 2014-01-07 17:44:06.386 31855 TRACE keystone.common.wsgi OperationalError: (OperationalError) (2003, "Can't connect to MySQL server on '10.35.160.66' (110)") None None
15:46:48 <kashyap> flaper87, Heya, I remember discussing this previously, but I forget. Do you know where can I find the list of arbitrary key/value pairs to be used w/  "glance image-update" --property <key=value>
15:46:49 <ayoung> so the server is up and running
15:46:56 <ayoung> but Keystone can't talk to it
15:47:03 <ohochman> morazi: you can say the administrator should know better .  but really how can he ?
15:48:17 <larsks> kashyap: Not sure, but http://docs.openstack.org/user-guide/content/glance-image-list.html shows several of them.
15:49:03 <Dominic> ohochman: that's a bug in the puppet manifests then, if they change a property and the service no longer works
15:49:20 <kashyap> larsks, Thanks, so much of docs! I'm looking for qemu guest agent in specific, I can't find it in the above URL
15:50:22 <kashyap> larsks, That page doesn't even list elementary hardware properties for Glance, like: hw_disk_bus, hw_cdrom_bus
15:52:26 <afazekas> f20+rdo has MULTIPLE selinux related issues, one of them causing the root-wrap is not working
15:54:39 <ohochman> Dominic: In openstack you have some key parameters such - mysql_ip, qpid_ip , etc.. that by changing them to the entire group belong on the same hostgroup will make problems for sure, not saying collapse the cloud ..   because even if it does change the services .conf parameters , the services might still need restart to change the actual configuration.
15:55:05 <Dominic> ohochman: puppet's entirely capable of restarting services
15:56:29 <ohochman> Dominic: I can see why it's consider a "feature"  I still think there should be other safe way to change multiple hosts params.
15:58:35 <ayoung> ohochman, morazi OK, so that machine is misconfigured, I think.  There is a working ,mysql DB populated on it, but the keystone server is looking for a mysql server on a different IP address
15:58:51 <ndipanov> kashyap, hey  u there?
15:58:57 <kashyap> ndipanov, Yes.
15:59:19 <ayoung> mysql://keystone:<uuid>@10.35.160.66/keystone
15:59:53 <ndipanov> kashyap, hey - my networking seems busted on fedora... am I missing something obvious?
16:00:13 <yfried> oblaut: cross tenant passes on rhel! No SecGroup Bug
16:00:31 <yfried> oblaut: where do I log this?
16:00:38 <kashyap> ndipanov, Ok, several possible reasons. I don't know what all you're using
16:01:21 <kashyap> ndipanov, However, if you'd like me to take a fuller look, you have to provide some info by running these two scripts:
16:01:33 <ndipanov> kashyap, hit me
16:01:37 <giulivo> does anyone know where I can find the .src.rpm files for icehouse?
16:01:48 <kashyap> ndipanov,     $ wget https://raw.github.com/larsks/neutron-diag/master/gather-network-info https://raw.github.com/larsks/neutron-diag/master/gather-neutron-info
16:01:49 <oblaut> yfried:  what do you mean ?  is it fixed in RDO ?
16:01:57 <yfried> I guess it didn't port
16:02:00 <yfried> don't know
16:02:03 <yfried> oblaut: ^
16:02:12 <kashyap> ndipanov, Instructions I noted here: lines 34-40 -- https://etherpad.openstack.org/p/rdo-testday-JAN2014
16:02:30 <ndipanov> vnice
16:02:37 <oblaut> yfried: does it mean RDO is not synced with Ice-House ?
16:02:41 <kashyap> ndipanov, I presume you're using Neutron? (Maybe no, you use nova-networking)
16:02:49 <yfried> oblaut: also reassociate FLIP and dns update works
16:02:54 <yfried> oblaut: no idea
16:03:20 <ndipanov> kashyap, neutron of course
16:03:55 <kashyap> ndipanov, Cool!
16:04:15 <yfried> oblaut: where should I log this?
16:04:40 <kashyap> ndipanov, That script will generate tar balls, you have to untar & SCP them somewhere. (If you have FAS account, fedorapeople.org would be easy)
16:06:03 <kashyap> ndipanov, Obvious things:  $ systemctl | grep neutron -- you see all of them ACTIVE?
16:06:46 <ndipanov> kashyap, lol no
16:06:53 <kashyap> :-)
16:06:54 <ndipanov> I told you it was something stupid
16:07:02 <ayoung> morazi, I'm done with that machine
16:07:40 <morazi> ayoung, ya that leads me to believe that either foreman got confused somewhere along the way due to the restart or there is something gnarly going on in the underlying puppet modules
16:07:47 <morazi> ayoung, thx!
16:08:03 <ayoung> morazi, NP.  feel free to point other people at me that have similar problems
16:08:21 <kashyap> ndipanov, Note: if it's something related to psutil, you can take a look at the workaround I mentioned in the Description here: https://bugzilla.redhat.com/show_bug.cgi?id=1049235
16:08:41 <morazi> ayoung, great, thx!  I think we got there by a really loopy means, and think it would be hard to reproduce it but I appreciate the offer
16:08:55 <ayoung> morazi, keystone/auth problems in general.
16:09:12 <kashyap> ayoung, Heya, since you're here, unverified question: is PKI default now in Keystone?
16:09:21 <ayoung> kashyap, it better be.
16:09:23 <ayoung> You tell me.
16:09:35 <ayoung> what does your keystone config file say kashyap ?
16:10:02 <kashyap> Hm, this returns 1 on my minimal setup: $ grep -i pki /etc/keystone/keystone.conf | grep -v ^$ | grep -v ^#
16:10:58 <kashyap> ayoung, I'm not using PKI on my setup. And, this is hand-configured. So, I can go to hell & not ask you any questions on this, I guess.
16:11:30 <kashyap> ayoung, My next new setup & all further will be PKI!
16:12:10 <ayoung> kashyap, let me look at what that old machine said
16:13:24 <ayoung> kashyap, so, there are 2 answers.  1st, yes, upstream PKI has been the default for a long time.  Second, for RDO, it looks like PKI is the default.  Tyhe machine I just looked at has
16:13:34 <ayoung> token_format =PKI
16:13:59 <ayoung> which is, I think, the deprecated way to indicate that the token provider should be the PKI provider....
16:14:51 <ayoung> kashyap, and , if you look in the [token] section, you will see that there is no explicit value for provider
16:15:21 <kashyap> ayoung, You're right.
16:16:21 <kashyap> ayoung,  Last interruption: do we have useage of Mozilla NSS here or is this all OpenSSL/something else?
16:16:33 <ayoung> still OpenSSL for now.  We'
16:16:37 <ayoung> re working on it
16:18:09 <kashyap> Cool, just being selfish - so I can dig my memory for NSS related tribal info :-) I recall jdennis is also working on it? Thx for the response
16:19:16 <ayoung> jdennis is who I meant be "we"
16:19:36 <kashyap> :-)
16:25:36 <kashyap> larsks, Found it here - http://docs.openstack.org/trunk/config-reference/content/kvm.html
16:29:06 <oblaut> Hi , what does packstack does when EPEL=n ?
16:29:42 <kashyap> Just guessing: It won't use/add any EPEL repositories
16:45:42 <jruzicka> vladan, if you tested Fedora 20 allinone, why not put it http://openstack.redhat.com/TestedSetups_2014_01 ?
16:49:58 <kashyap> pixelb, Can I move this bug to RDO component - https://bugzilla.redhat.com/show_bug.cgi?id=981116
16:50:39 <kashyap> pixelb, But sometimes I like the usefulness of Fedora Updates System, which closes the bug automatically for us once the update goes to stable (comment #30).
16:50:43 <pixelb> kashyap, it really is a Fedora/systemd issue
16:51:31 <kashyap> pixelb, Gah, I'm looking too much at screen. I totally forgot that I filed the bz myself. Ignore me!
16:51:50 <pixelb> I'll log systemd bug anyway
16:53:19 <pixelb> Heh already done: https://bugzilla.redhat.com/1014311
16:57:06 <pixelb> larsks, That particular systemd symlink issue you were asking about is https://bugzilla.redhat.com/1014311
16:57:23 <kashyap> pixelb, Should I add "Depends On" field for 981116 pointing to 1014311?
16:57:32 <pixelb> kashyap, done
16:58:05 <kashyap> Ah, I didn't refresh my browser :-) Thx
17:01:28 <kashyap> ndipanov, Able to progress on Neutron problems?
17:01:37 <ndipanov> kashyap, no
17:01:47 <ndipanov> still can't get network to work
17:02:08 <kashyap> ndipanov, SELinux permissive? (For now)
17:02:40 <kashyap> Also, there are a couple of issues lurking around (rootwrap SELinux issues, etc). Once I have something more concrete, I'll post you details.
17:03:06 <ndipanov> kashyap, yeah
17:03:23 <ndipanov> kashyap, hit another bug with volumes
17:03:28 <ndipanov> but want to sort this out first
17:04:50 <kashyap> Sure.
17:05:11 <DrBacchus> Ah. AMQP isn't running. Bah.
17:05:42 <DrBacchus> AMQP, that is
17:05:58 * DrBacchus squints
17:26:27 <afazekas> kashyap,ndipanov audit log added to the https://bugzilla.redhat.com/show_bug.cgi?id=1049503
17:28:43 <ndipanov> afazekas, I saw the same thing when booting with volumes
17:28:51 <ndipanov> wel similar
17:29:19 <larsks> pixelb: Thanks for the link...
17:30:03 <kashyap> afazekas, I saw that, added a comment there.
17:30:20 <afazekas> ndipanov: the selinux policies has multiple issues more than 3 IMHO
17:30:49 <ndipanov> kashyap, did you see any neutron issues with selinux?
17:31:25 <kashyap> ndipanov, I don't see anything when I tried to generate a reference policy: $ cat /var/log/audit/audit.log | audit2allow -R
17:31:31 <afazekas> on f20 I saw, on RHEL 6.x it seams to be ok
17:31:47 <ndipanov> afazekas, with neutron?
17:33:04 <afazekas> With neutron the first visible thing , you do not have namspaces
17:33:24 <afazekas> almost all ip related command fails with permission denied
17:34:18 <ndipanov> afazekas, due to selinux?
17:34:19 <kashyap> ndipanov, Do you see anything with: $ cat /var/log/audit/audit.log | audit2allow -R
17:34:58 <afazekas> ndipanov: rootwrap related avc denials are in the audit.log, so yes
17:35:23 <ndipanov> afazekas, yes silly me
17:44:20 <ndipanov> afazekas, kashyap are you guys gonna report those
17:45:00 <kashyap> ndipanov, I don't have the SELinux errors handy as of now, I just turned SELinux to Enforcing (from permissive), cleared my logs, & trying to capture these.
17:45:19 <ndipanov> kashyap, will do the same
17:45:19 <kashyap> (Permissive should still have captured these things in audit.log, don't know why it didn't)
17:45:29 <ndipanov> but I had a ton of them
17:45:52 <afazekas> ndipanov: I am not sure we really want to report all selinux issue individually
17:46:22 <kashyap> $ cat /var/log/audit/audit.log | audit2allow -R
17:46:23 <kashyap> could not open interface info [/var/lib/sepolgen/interface_info]
17:46:25 <ndipanov> afazekas, that's why I was asking
17:46:27 * kashyap first tries to resolve that
17:46:43 <kashyap> afazekas, You can report them all together by running the above audit2allow -R command
17:48:10 <afazekas> kashyap: last time when I reported selinux issue the  audit2allow was not considered as enough detail
17:49:13 <kashyap> afazekas, I mean, full audit.log is always useful, but just noting my previous experience of resolving an issue w/ it  :-)
17:50:23 <afazekas> yes, the audit2allow output is usually more human readable ;-)
17:51:24 <kashyap> :-)
17:57:14 <jbernard> if anyone is seeing strange output with 'cinder rate-limits', this bug (https://bugzilla.redhat.com/show_bug.cgi?id=1037728) is still present
18:04:58 <kashyap> jbernard, I'm not a Cinder guy; but, an obvious question: are you hitting this w/ all latest cinder packages?
18:08:10 <ndipanov> kashyap, btw - I had to enable both 80 and 6080 in firewalld by hand to get to horizon/vnc
18:08:28 <ndipanov> kashyap, that's a bug right?
18:08:54 <kashyap> ndipanov, I'm afraid, I don't have Horizon in my setup; so I can't confirm.
18:08:58 <kashyap> mrunge, ^^ ?
18:09:28 <kashyap> or jpich; if you have a moment (for the above question)
18:09:32 <ndipanov> kashyap, well it's not really a mrunge bug
18:09:52 <ndipanov> well it is a bit actually :)
18:11:03 <kashyap> Yeah, just that since they're more in the trenches with these; maybe they (he/jpich) know such stuff top off their heads
18:13:12 <weshay> pixelb, ya.. the jobs are now completing on their first try of the packstack install.. so the wrkarounds are not used :)
18:13:51 <shardy> Anyone know if this ceilometer-api failure has an existing bug?
18:13:54 <shardy> http://fpaste.org/66476/13891181/
18:14:08 <pixelb> weshay, great stuff. The CI speeded things up hugely for me last night. I had multiple things testing in parallel
18:14:21 * shardy couldn't find one
18:14:24 <weshay> great!! that is what I want to hear
18:14:40 <jpich> kashyap: I'm afraid I'm unable to check anything right now, sorry
18:15:09 <pixelb> shardy, not seen that. Do you have python-wsme-0.5b6 installed?
18:16:15 <shardy> pixelb: No, I this is RDO Havana, installed via F20 repos, so I ended up with python-wsme-0.5b2-2
18:16:29 <shardy> pixelb: Do I need to enable updates-testing?
18:16:35 <jbernard> kashyap: icehouse-1
18:16:47 <pixelb> shardy, Ah I thought you were on icehouse testday stuff
18:17:09 <shardy> pixelb: I wanted to test Havana first, then Icehouse on a second machine
18:17:16 <jbernard> kashyap: the 'prettytable' package is too old
18:17:31 <jbernard> kashyap: or, the cinder client is too new, depending on how you look at it
18:18:00 <pixelb> shardy, python-wsme-0.5b5 is in updates-testing, so it would be worth testing with that
18:18:00 <shardy> pixelb: unfortunately the Havana testing has not gone as smoothly as I'd hoped :(
18:18:40 <shardy> pixelb: Ok, thanks, will give it a try
18:20:13 <kashyap> jpich, No worries.
18:21:06 <kashyap> giulivo, You have any hints for the bug jbernard is encountering? (I see you were part of the bz he referenced)
18:22:32 <shardy> pixelb: Thanks, that fixed it
18:22:47 <pixelb> shardy, OK cool. That's already sumitted for stable
18:22:57 <jbernard> kashyap, giulivo: if it's possible to bump the version of 'python-prettytable' package from 0.6.1 to ≥ 0.7, that would be ideal (assuming it doesn't break anything else)
18:24:26 <ndipanov> pixelb, kashyap I see this iscsiadm: Could not execute operation on all records: no available memory when booting with volume even with permissive on - anyone else seen it?
18:24:30 <kashyap> jbernard, Reason? New upstream release? Or other bug-fix?
18:24:59 <pixelb> ndipanov, not seen it sorry
18:25:12 <jbernard> the current version of 'python-cinderclient' (1.0.7) requires ≥ 0.7 of python-prettytable
18:25:30 <mrunge> sorry, I was on a call
18:25:39 <mrunge> kashyap, ndipanov what was(is the issue?
18:25:50 <kashyap> shardy, A qucik note: If you want to quickly find the latest build available in Koji for a specific package: koji latest-build rawhide python-wsme
18:26:06 <ndipanov> mrunge, I had to manually open 80/6080 ports after neutron restart
18:26:11 <kashyap> shardy, Couple related stuff here: http://openstack.redhat.com/DeveloperTips
18:26:22 <pixelb> jbernard, prettytable has prettyterrible backwards compat history, so we'll need to be careful
18:26:25 <kashyap> ndipanov, Haven't seen your issue (yet).
18:26:48 <jbernard> pixelb: im hearing the same thing from a few folks
18:26:56 <shardy> kashyap: Ok, thanks :)
18:26:58 <japerry> gmorning -- somewhat new to RDO/Openstack but was going to try to install icehouse after my failed attempt with FC19 andGrizzly
18:27:10 <ndipanov> japerry, good luck
18:27:10 <pixelb> jbernard, are you getting a runtime issue? or did you just observe this in requirements.txt?
18:27:22 <mrunge> ndipanov, I have seen so many strange things with neutron on fedora
18:27:31 <kashyap> shardy, Np;  just noting them explicitly as people are already buried in a gazillion URLs
18:27:37 <ndipanov> mrunge, thanks that makes me feel better
18:27:37 <mrunge> ndipanov, I'm not sure where to start there....
18:27:47 <jbernard> pixelb: installed with packstack, the current pairing of cinderclient-to-prettytable is mismatched
18:27:53 <japerry> thanks ndipanov! I'll report back any issues.. its a brand new updated FC20 install
18:27:54 <ndipanov> mrunge, but yeah - I could not get it to work afaicr
18:28:23 <jbernard> pixelb: pip install prettytable fixes the cinderclient issue, but it may cause regressions elsewhere
18:28:47 <pixelb> jbernard, so you get a runtime failure without doing that?
18:29:02 <jbernard> pixelb: yes, the expected output is missing
18:29:19 <pixelb> heh ok, jruzicka ^^
18:29:42 <jbernard> pixelb: i just prints nothing, so the it not entirely obvious that it's failing
18:29:51 <jbernard> pixelb: unless you know the expected behaviour
18:30:32 <jbernard> im happy to help test different versions of either package, or anything else that would help
18:31:06 <kashyap> afazekas, For later, this also gives a human-readable error: $ audit2allow -w -a
18:33:58 <pixelb> jbernard, could you rpm -e python-cinderclient; yum install http://rdo.fedorapeople.org/openstack-havana/epel-6/python-cinderclient-1.0.6-2.el6.noarch.rpm
18:34:07 <pixelb> (to see if previous version is OK)
18:37:19 <jbernard> pixelb: same behaviour with 1.0.6-2
18:40:51 <pixelb> jbernard, Hmm, I reviewed the upstream 0.6 -> 0.7 changes and they look sensible enough, so I'll see if I can provide an update quickly...
18:42:17 <jbernard> pixelb: awesome, im happy to test it
18:46:30 <zaitcev> "No package opnstack-packstack available." grrrr
18:47:34 <jbernard> zaitcev: spelling mistake?
18:47:43 <jbernard> zaitcev: ahh, you knew that ;)
18:48:30 <zaitcev> Sorry for the noise, but I was this close to tweet it. Very silly, yes.
18:53:11 <afazekas> it does not seems good: http://www.fpaste.org/66492/13891207/
19:12:40 <afazekas> http://www.fpaste.org/66498/91217481/ do I need to enable something to get the dhcp agent bindings ?
19:12:52 <pixelb> jbernard, Could you try: yum install http://kojipkgs.fedoraproject.org/packages/python-prettytable/0.7.2/1.el6/noarch/python-prettytable-0.7.2-1.el6.noarch.rpm
19:14:43 <jbernard> pixelb: same behaviour
19:15:29 <pixelb> hrm. That's surprising since that's the version pip would be installing
19:16:02 <jbernard> pixelb: oh hang on, i had a typo
19:16:05 <jbernard> pixelb: one sec
19:17:19 <larsks> I'm just catching up on things.  Has anyone else seen a "Could not parse for environment production" when applying xxx_neutron.pp?
19:17:40 <kashyap> afazekas, What are you running? F20?
19:18:23 <jbernard> pixelb: that works
19:18:30 <larsks> Hmmm, might be a bad config.  Checking.
19:18:30 <jbernard> pixelb: output is now correct from cinderclient
19:18:43 <pixelb> jbernard, OK thanks for testing! Copying to repos now..
19:18:51 <jbernard> pixelb: awesome!
19:19:20 <jbernard> pixelb: i dont think it should cause any troubles with other packages, but ill keep an eye out
19:20:06 <DrBacchus> Different from "Failed to parse /etc/nova/nova.conf" ?
19:22:45 <afazekas> kashyap: tempest-packstack scenario test
19:23:03 <DrBacchus> I'm getting "No valid host was found" when I try to launch an instance. Where should I be looking?
19:24:56 <afazekas> kashyap: on the f20 cloud image, disabled selinux, iptables, firewalld, applied nova-libvirt patch,  mysql-mariadb workaround  with neutron
19:25:49 <kashyap> afazekas, Ok, I'll check in a few, taking a break - been staring at a screen for a while.
19:32:28 <afazekas> http://www.fpaste.org/66503/12312313/
19:33:18 <afazekas> cinder backup.log
19:34:00 <giulivo> afazekas, something wrong with the db, probably empty
19:34:11 <giulivo> I think Dafna had that earlier today but we couldn't reproduce
19:35:29 <afazekas> May be I got the error t install time, now it is there
19:37:02 <morazi> eharney, ^^ you know anyone who could take a look there?
19:41:48 <jbernard> afazekas: does 'cinder-manage db sync' help?
19:42:13 <jbernard> afazekas: probably not…
19:46:07 <afazekas> jbernard: I do not see issue with service yet, It just added it to the log probbaly at the startup time
19:47:48 <afazekas> cinder backup-create b54fa509-dfa8-4550-b5a8-5f3df1132d54
19:47:48 <afazekas> ERROR: Service cinder-backup could not be found. (HTTP 500) (Request-ID: req-eb54fecf-e9bf-40a0-94f3-d857773fa9d1)
19:49:11 <jbernard> afazekas: i don't think the packstack cinder is configured to start cinder-backup
19:49:26 <jbernard> afazekas: i would guess you'll not see 'cinder-backup' in your process list
19:49:31 <afazekas> jbernard: the server was not ran
19:49:46 <afazekas> after starting it the request was accepted
19:50:20 <afazekas> would be nice if the openstack-status would include this service
19:51:44 <afazekas> is it possible the packstack tried to start this service before the db sync ?
19:52:26 <jbernard> i agree
19:52:34 <jbernard> on that, i am not sure
19:53:11 <jbernard> the backup will fail if the service is not available to handle the request (as I understand)
19:53:21 <jbernard> that could be a friendlier message
19:53:40 <jbernard> and i think the exception you're seeing is probably a different issue
19:53:51 <afazekas> I think the service should be enabled and running after the packstack install, so it is bug anyway
19:54:03 <jbernard> i agree with you there
19:54:31 <afazekas> It is time to live , bye
19:54:46 <DrBacchus> afazekas: Thanks for helping out!
20:37:08 <paul__> hello
20:37:35 <paul__> i have a probel with RDO and openvswitch, anyone can help ?
20:37:53 <DrBacchus> Hi, paul__
20:37:59 <DrBacchus> Can you elaborate?
20:39:20 <paul__> sure
20:39:34 <paul__> I can't join my vm on openstack even from compute node itself
20:39:59 <paul__> I assume it is a problem with openvswitch config but nor sure
20:40:12 <paul__> ovs-vsctl show
20:40:14 <paul__> ed54db06-7cb7-415a-97ea-2bc2ef85e085
20:40:15 <paul__> Bridge br-int
20:40:17 <paul__> Port "qvo4f65df07-8e"
20:40:18 <paul__> tag: 3
20:40:20 <paul__> Interface "qvo4f65df07-8e"
20:40:21 <paul__> Port br-int
20:40:23 <paul__> Interface br-int
20:40:24 <paul__> type: internal
20:40:26 <paul__> Bridge br-ex
20:40:27 <paul__> Port br-ex
20:40:29 <paul__> Interface br-ex
20:40:30 <paul__> type: internal
20:40:32 <paul__> ovs_version: "1.11.0"
20:40:33 <paul__> the vm is interface qvo4f...
20:42:02 <DrBacchus> And you can't ssh to it? ping it?
20:43:34 <paul__> nothing
20:43:45 <paul__> (even if rules added in security group)
20:45:57 <DrBacchus> Can anyone help paul__ out with a networking-related problem?
20:46:34 <larsks> paul__: Are you running an all-in-one configuration? Or do you have separate compute and network hosts?
20:47:12 <paul__> all-in-one
20:53:35 <larsks> paul__: Can you post somewhere the outputs of "nova list", "neutron net-list", "neutron router-list", and "neutron router-port-list <router>" for each router?
20:53:42 <paul__> what should be a working configuration ? ovs-vsctl result ?
20:53:43 <larsks> ...and also "neutron subnet-list"
20:54:45 <larsks> Also, if there are any rdo test day people still around, was there any traffic regarding neutron and selinux?  because icehouse neutron on my f20 system is super unhappy.
20:55:05 <Egyptian[Laptop]> paul__: what is your br-ex interface config like?
20:55:51 <DrBacchus> I didn't see any selinux traffic today.
20:58:15 <DrBacchus> Looks like kashyap and ndipanov discussed some selinux/neutron issues a few hours ago.
20:58:43 <DrBacchus> But it doesn't look like there was much detail.
21:00:45 <larsks> DrBacchus: Thanks.  Bug time I guess.
21:09:24 <Egyptian[Laptop]> is there a workaround for this? Error: /Stage[main]/Ceilometer::Db/Exec[ceilometer-dbsync]: Failed to call refresh: ceilometer-dbsync --config-file=/etc/ceilometer/ceilometer.conf returned 1 instead of one of [0]
21:09:47 <Egyptian[Laptop]> ah its a mongodb thing
21:13:08 <Egyptian[Laptop]> i used the following parameters : packstack --allinone --nagios-install=n --mysql-pw=password --ntp-servers=0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org --os-swift-install=y --provision-demo-floatrange=192.168.0.0/24 --keystone-demo-passwd=password     ... and packstack configured my br-ex to take the same ip as my gateway
22:19:52 <zaitcev> yay, finally progressed deep enough to hit known workarounds
01:50:37 <japerry> ping, anyone here testing Fedora 20 w/ Neutron and packstack?
01:50:53 <japerry> with the icehouse install, from stock I cannot seem to remove the network included
06:34:16 <mbourvin> good morning
06:34:37 <mbourvin> I was wondering - why does it take so much time to delete an empty volume that I just created?
06:43:02 <mbourvin> abaron: ewarszaw: ^^
06:44:02 <abaron> mbourvin: if it is 'post zero' then it is because we write zeros on it
06:44:42 <mbourvin> abaron: how do I know if it's `post zero` or not?
06:45:10 <abaron> simplest way of knowing whether the delete was post zero is to look at the vdsm log
06:45:14 <abaron> at the delete operation
06:45:21 <abaron> and check to see if postZero param is true
06:45:44 <mbourvin> abaron: vdsm? is there a vdsm in openstack that I don't know about? ;)
06:46:02 <abaron> mbourvin: lol
06:46:09 <abaron> mbourvin: same issue though
06:46:17 <abaron> mbourvin: safe delete writes zeros
06:46:49 <abaron> mbourvin: ping flaper87|afk to know what exactly to look for
06:47:08 <mbourvin> abaron: so what is the max time that you think that it should take for deleting a 15G volume for example?
06:47:35 <abaron> mbourvin: as long as it takes to write 15G of zeros on your storage (it depends on the storage)
06:47:47 <mbourvin> abaron: local lvm?
06:48:41 <abaron> mbourvin: test it.  run 'time dd if=/dev/zero of=/dev/VGNAME/LVNAME'
06:49:03 <abaron> mbourvin: are you using the thin lvm?
06:49:09 <mbourvin> abaron: nope
06:49:19 <abaron> then test the above command
06:49:27 <abaron> or create smaller volumes
06:50:33 <abaron> note that here too, the only place in automatic tests that you want to call delete volume from the API is in the 'delete volume' test.  Everwhere else you want to have your teardown do it in a much faster and dirtier way
06:51:34 <mbourvin> abaron: ok, but now I'm running manual tests, and creating big volumes is a part of it :)
06:53:10 <abaron> then cinder has a configuration option to disable safe delete
06:53:11 <abaron> https://github.com/openstack/cinder/commit/bb06ebd0f6a75a6ba55a7c022de96a91e3750d20
06:53:19 <abaron> mbourvin: ^^^
06:54:19 <mbourvin> abaron: ok, thanks, anyway I see the dd commend that it runs in top, and you were right - this is why it takes so much time
06:54:31 <mbourvin> *command
07:08:50 <mbourvin> abaron: I'm thinking - maybe the default should safe delete - disabled
07:09:19 <mbourvin> abaron: It's weird that it takes so much time to delete a volume that was just created (from the user point of view)
07:10:20 <abaron> the default should be as it is.  with thin target it will be a lot faster, but with thick it is raw and we have no knowledge of where user information resides
07:10:39 <abaron> this means that the only safe option is to zero it all before exposing the same disk to another user
07:10:47 <abaron> otherwise data leakage will occur
07:11:17 <abaron> users need to consciously decide that they don't care about their data falling into other users' hands
07:11:37 <abaron> users also don't care how long it takes to delete
07:12:03 <abaron> the only ones who care are admins / and quality engineers who rely on it for their next tests ;)
07:13:18 <mbourvin> abaron: lol ok :)
07:42:54 <mbourvin> I have a new issue - I created a big volume and tried to delete it - but there is no space left on device now. Horizon is dead, and when I run `cinder list` "I got this error: ERROR: An unexpected error prevented the server from fulfilling your request. (OperationalError) (2003, "Can't connect to MySQL server on '10.35.64.87' (111)") None None (HTTP 500)"
07:43:42 <mbourvin> shouldn't it prevent creating such a big disk if there is not enough space? maybe like it works in ovirt?
07:43:48 <mbourvin> abaron: yrabl: ^^
07:48:56 <kashyap> Morning
08:28:10 <kashyap> ndipanov, Heya, Have you filed the issue for the rootwrap issues?; If not, I'm just drafting a report
08:28:21 <kashyap> I presume it's this:
08:28:23 <kashyap> 2014-01-08 03:01:34.484 25113 TRACE neutron.agent.dhcp_agent Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'link', 'set', 'tap2816cd25-73', 'address', 'fa:16:3e:7b:f8:9e']
08:32:57 <kashyap> Hm, I see something else - an IOError of some sort:
08:32:57 <kashyap> 2014-01-08 03:31:24.172 1713 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.7/site-packages/eventlet/hubs/poll.py", line 44, in register
08:32:57 <kashyap> 2014-01-08 03:31:24.172 1713 TRACE neutron.agent.dhcp_agent     self.poll.register(fileno, mask)
08:32:57 <kashyap> 2014-01-08 03:31:24.172 1713 TRACE neutron.agent.dhcp_agent IOError: [Errno 22] Invalid argument
08:33:02 * kashyap investigates further
08:33:25 <ndipanov> kashyap, yes - plus I got one with iscsiadm
08:33:35 <ndipanov> kashyap, but that one seems to be not related to selinux
08:33:45 <ndipanov> so I'm investigating that
08:34:07 <kashyap> ndipanov, Ok, if you filed, can you please point me to bz URL?
08:34:37 <ndipanov> kashyap, not yet - want to make sure it's not my system
08:38:57 <kashyap> Ok, I'm certain of a bunch of SELinux issues, just getting the relevant logs to dig further. Thanks.
08:40:12 <ohochman> ayoung: ping
08:40:35 <kashyap> ohochman, ayoung is in Boston, and asleep.
08:41:07 <ohochman> kashyap: oh,
08:47:18 <kashyap> If you have a Keystone question, just post here, someone who's more familiar might take a look
08:51:06 * flaper87 is here
08:56:04 <kashyap> ndipanov, When you get a sec, can you please post this -- $ cat /var/log/audit/audit.log | audit2allow -R
08:56:08 <kashyap> I just wanted to compare
08:56:21 <anand_> I have installed openstack using rdo in a vm in esx, I need to add a compute node to this setup.how to do that
08:56:44 <kashyap> http://openstack.redhat.com/Adding_a_compute_node
08:56:52 <ndipanov> kashyap, sure will do - let me clean it first and start over
09:21:48 <kashyap> ndipanov, FYI, I filed this  - https://bugzilla.redhat.com/show_bug.cgi?id=1049807
09:23:07 <ndipanov> kashyap, very nice - want to maybe add those critical (ish) bugs to notes?
09:23:21 <kashyap> ndipanov, You mean - workaround?
09:23:42 <ndipanov> well more like - have them there in one place
09:23:44 <kashyap> (s/workaround/workarounds wiki page)
09:23:54 <kashyap> Yes, adding there.
09:26:06 <ndipanov> kashyap, we already have setenforce 0 in workarounds
09:26:09 <ndipanov> :)
09:27:40 <kashyap> Yeah, next time, we (I) should post more details so folks can try to post relevant SELinux failures (with *useful* investigative info)
09:40:03 <ndipanov> kashyap, but the crux of the issue is selinux policy is broken for neutron
09:40:23 <kashyap> ndipanov, Yes, I'm talking to SELinux Fedora maintainers & debugging w/ him
09:40:52 <ndipanov> kashyap, comparing your audit2allow to mine now
09:52:47 <ndipanov> kashyap, might be good to add fedora 20 to that bug (though clear from the nvrs)
09:53:16 <kashyap> ndipanov, Done, sir. It's for selinux-policy component - https://bugzilla.redhat.com/show_bug.cgi?id=1049817
09:53:23 <kashyap> Upstream is looking into it
09:53:59 <kashyap> Changed the version from rawhide to F20
09:54:40 <ndipanov> kashyap, fwiw your denials are more comprehensive than mine
09:54:43 <ndipanov> but still comparing
09:55:43 <ndipanov> kashyap, btw is that output more useful then doing just audit2allow -a ?
09:56:35 <kashyap> ndipanov, Well,  -R just allows us to generate a reference policy to make things working.
09:56:54 <kashyap> ndipanov, $ audit2allow -a -w  gives a more descriptive/human-readable detail
09:57:05 <ndipanov> kashyap, hence my question
09:57:07 <ndipanov> :)
09:57:50 <kashyap> Yes, that'd be useful. There'll be a build later today in Koji, mgrepl (Fedora SELinux maintainer) just confirmed
09:58:29 <ndipanov> kashyap, awesome
10:04:51 <Dafna> tshefi_: hows it going? any bugs this morning?
10:04:59 <Dafna> giulivo: ^^
10:07:48 <tshefi_> Dafna, none yet but now on my cc im getting such syslogs cougar01 iscsid: conn 0 login rejected: initiator error - target not found (02/03)
10:11:13 <tshefi_> Dafna, not sure why as my cinder and glance are on other hosts.
10:12:27 <Dafna> tshefi_: are you sure that you installed the ice-house packages?
10:12:46 <Dafna> tshefi_: what is your packstack?
10:13:01 <tshefi_> Dafna, one sec
10:13:36 <tshefi_> Dafna, which component do you want to know?
10:13:52 <Dafna> tshefi_: rpm -qa |grep packstack
10:14:34 <tshefi_> Dafna, openstack-packstack-2013.2.1-0.25.dev936.el6.noarch
10:16:23 <pixelb> tshefi_, that should be 0.27... ?
10:16:27 <giulivo> Dafna, I was having troubles with F20 but abandoned it so I'm moving on thinlvm on rhel
10:16:55 <pixelb> tshefi_, that might suggest you have a havana repo rather than icehouse?
10:17:26 <Dafna> giulivo: what happened with the emc?
10:17:42 <giulivo> got a couple of bugs open
10:17:51 <tshefi_> pixelb, how could that be? Ran setup from RDO quick guide step by step?
10:18:02 <giulivo> but I'm done with it
10:18:10 <tshefi_> pixelb, check my repo folder now
10:20:11 <giulivo> Dafna, one thing though, we don't configure cinder for backups ... it's not just about starting the service, it's unconfigured
10:20:20 <giulivo> did you try to get volume backups?
10:21:27 <tshefi_> pixelb, correct checking rdo-release.repo says Havana grrrrrr!!! but i used the RPM from RDO site after it was corrected how come i still got Havana?
10:22:04 <pixelb> tshefi_, You followed step 1 at http://openstack.redhat.com/RDO_test_day_January_2014#How_To_Test
10:22:39 <pixelb> ? There should be no other instructions anywhere to confuse things for icehouse testing
10:23:02 <tshefi_> pixelb, yeah sure my hosts were blank no repos from before, used /rdo-release-icehouse.rpm  for sure
10:25:00 <tshefi_> pixelb, i'll test on a new host using icehouse-rpm (again) and check the rdo. repo for icehouse version, be back soon thanks.
10:25:13 <Dafna> tshefi_: lets take this off line
10:25:49 <pixelb> tshefi_, It should all be recorded. yum history list rdo-release; yum history info $id
10:26:19 <Dafna> giulivo: so emc is blocked for testing in ice-house?
10:26:30 <giulivo> no it's not blocked, it's done
10:26:47 <Dafna> giulivo: so it worked?
10:26:59 <giulivo> yeah part of the features work, others don't
10:27:17 <giulivo> I'm now drafting a bug for the backup functionalities which doesn't affect only the emc driver
10:27:21 <giulivo> and then move to something else
10:28:14 <Dafna> giulivo: open bugs for everything that doesn't work
10:28:28 <Dafna> giulivo: do it before you move on...
10:28:42 <giulivo> sure
10:31:29 <nmagnezi> kashyap, thanks for checking that SElinux issue. plz let me know whan it's resolved and i'll re-run my tests with Enforcing mode.
10:32:59 <kashyap> nmagnezi, Np. It should be in build system later today. You may have to pull it from build system.
10:33:15 <Dafna> giulivo: thank you
11:09:39 <Dafna> giulivo: ping
11:09:47 <giulivo> Dafna, pong
11:10:08 <Dafna> giulivo: should a volume be stateless when I boot an instance from it?
11:10:14 <giulivo> no in-use
11:11:02 <Dafna> giulivo: no... I mean, when I boot an instance from a volume -> create a file -> shut down the instance -> boot a second instance from the volume -> should I see the file?
11:11:22 <giulivo> Dafna, yes, definitely
11:11:29 <Dafna> giulivo: mmm
11:11:44 <giulivo> you should see the file, it's not stateless, the whole point of volumes is in data *not* being stateless
11:11:45 <Dafna> giulivo: maybe because it was on /tmp?
11:11:51 <giulivo> maybe yes
11:12:11 <Dafna> giulivo: bug still... but I will try to create a dir under root
11:12:39 <giulivo> oh tmp is probably cleaned up by the OS, cinder isn't touching the data in the volumes
11:20:04 <anand_> error in adding multinode. it stops while running packstack-answerfile in host1
11:25:16 <shardy> Is anyone else seeing -ECONNREFUSED qpid errors after a reboot (F20, Havana)?
11:26:23 <shardy> nova, ceilometer, heat and neutron can't connect until I restart the qpidd
11:27:45 <afazekas> Where is the epel-7 repo ?
11:28:51 <afazekas> http://repos.fedorapeople.org/repos/openstack/openstack-icehouse/epel-7/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found
11:40:15 <kashyap> afazekas, As near as I know, EPEL7 doesn't have all the packages generated
11:40:32 <kashyap> Compare that $ koji list-pkgs --tag=epel7 | wc -l  with  $ koji list-pkgs --tag=f20 | wc -l
11:40:55 <kashyap> shardy, Nope, haven't seen it on my Havana F20 setup (it's now IceHouse).
11:41:25 <kashyap> shardy, What does that say -- $ netstat -lnptu | grep qpid
11:41:45 <kashyap> I had to always restart qpidd though after reboot: $ systemctl restart qpidd
11:42:29 <shardy> kashyap: That's what I'm finding, restarting qpidd should not be necessary IMO, must be a bug somewhere
11:43:47 <shardy> I raised bug #1049488 yesterday, but I see dneary also raised a similar bug back in July..
11:45:56 <afazekas> kashyap: I'll try the f-20 packages..
11:47:37 <kashyap> shardy, Yeah, I missed to file it, thanks.
11:47:57 <mbourvin> is there a log collector?
11:49:17 <kashyap> shardy, Can you please also add dneary's bug in your bz? just as a reference
11:49:48 <mbourvin> ohochman: nmagnezi: kashyap: is there a log collector?
11:49:52 <kashyap> mbourvin, For Neutron, you can follow-- https://etherpad.openstack.org/p/rdo-testday-JAN2014 lines 34 to 40
11:50:03 <mbourvin> kashyap: and for others?
11:51:04 <yrabl> giulivo, ping
11:51:07 <kashyap> mbourvin, Needs to be made. Can you be more specific?  "log collector" is a vague phrase
11:51:08 * kashyap off to lunch
11:51:19 <ohochman> kashyap: where the foreman-section on : https://etherpad.openstack.org/p/rdo-testday-JAN2014
11:52:49 <shardy> kashyap: done
11:56:18 <dneary> kashyap, My bug?
11:56:56 <shardy> dneary: https://bugzilla.redhat.com/show_bug.cgi?id=984968
11:57:14 <shardy> Which I think I duped yesterday with bug #1049488
11:57:33 <shardy> dneary: Did you get any feedback, at all, as to why that is happening?
11:58:19 <shardy> seems like folks have all just been bouncing qpidd after reboot, which seems, uh, suboptimal
11:58:31 <dneary> shardy, No more than in the forum threads & wiki page I pointed at
11:58:47 <dneary> Definitely suboptimal
11:58:53 <dneary> Esp if it's occurring a lot
11:59:02 <dneary> Seems to be a Fedora thing, not CentOS/RHEL
11:59:07 <shardy> dneary: Ok, thanks, I'm trying to dig into it a bit as it's happening repeatably for me every reboot (fresh F20 with Havana)
11:59:40 <shardy> dneary: Yeah I don't think I saw it on EL6.5, but that was RHOS not RDO so too many variables
12:02:52 <kashyap> dneary, You got the context above.
12:04:30 <giulivo> pong yrabl
12:10:41 <shardy> Does anyone have any tips for what needs to be done to enable instances to access the nova metadata service with neutron networking?
12:11:02 <yrabl> giulivo, how is the smi-s?
12:11:11 <giulivo> it worked
12:12:15 <giulivo> yrabl, I have the RDO deployment still configured if you want to run further stuff there
12:12:56 <yrabl> giulivo, no, it's ok - I still have the glusterfs to handle :). just checking
12:16:38 <pradeep> i have RDO deployment. and bare metal provision always fails.
12:17:09 <pradeep> find multiple road blocks. Any one successfull  with bare metal with RDO?
12:32:13 <ohochman> pixelb: ping
12:39:14 * afazekas can't find ruby(selinux)  libselinux-ruby for epel-7
12:41:41 <Dominic> in EL6 it was part of the optional channel, not EPEL
12:41:41 <ohochman> Dominic: ping
12:41:41 <ohochman> hi
12:41:44 <Dominic> hi
12:42:10 <ohochman> Dominic: I have some new AVCs on neutron-controller deployed by foreman
12:42:26 <ohochman> avc:  denied  { read } for  pid=18313 comm="ip" path="/proc/18223/status" dev=proc ino=222926 scontext=unconfined_u:system_r:ifconfig_t:s0 tcontext=unconfined_u:system_r:initrc_t:s0 tclass=file
12:42:33 <ohochman> avc:  denied  { read write } for  pid=18527 comm="rsync" path="/tmp/puppet20140108-18223-5dcjp8-0" dev=dm-0 ino=524356 scontext=unconfined_u:system_r:rsync_t:s0 tcontext=unconfined_u:object_r:initrc_tmp_t:s0 tclass=file
12:43:16 <Dominic> file BZs then I guess, but it's not against Foreman
12:43:27 <ohochman> foreman-selinux /
12:43:28 <ohochman> ?
12:43:43 <Dominic> no, this isn't a Foreman host, not a Foreman bug
12:43:58 <ohochman> openstack-selinux
12:44:01 <ohochman> ?
12:44:09 <Dominic> yeah, probably
12:44:11 <ohochman> ok
12:44:16 <ohochman> thanks,
12:52:53 <ohochman> Dominic: I should have openstack-selinux installed on the machine right. if it's there none it's probably because the neutron-controller host group not including those puppet manifest that installing it
12:52:56 <ohochman> ?
12:54:19 <ohochman> pixelb: ?
12:59:42 <ndipanov> pixelb, hey - I seem to be hitting https://bugzilla.redhat.com/show_bug.cgi?id=1001705
13:20:44 <giulivo> ndipanov, I was expecting some upstream fix to be found in the RDO pkgs but the change is dated 6th of Dec, the RDOs are based on the 5th of Dec tarballs right?
13:21:15 <ndipanov> giulivo, you talkin bout the iscsi thing
13:21:16 <ndipanov> ?
13:21:29 <giulivo> no this one https://review.openstack.org/#/c/54833/
13:21:37 <ohochman> morazi: ping
13:21:45 <morazi> ohochman, pong
13:22:34 <ohochman> morazi: I notice that there's no openstack-selinux installed on the neutron-controller (and some AVCs remains after deployment)
13:22:40 <ndipanov> giulivo, so not related to my comment form a few mins back?
13:23:24 <ndipanov> from*
13:23:46 <giulivo> ndipanov, no it's not
13:23:48 <ndipanov> btw we base our packages on milestone releases
13:23:56 <ndipanov> it's not just a random commit
13:24:00 <ohochman> morazi: I've opened a bug on openstack-selinux but I actually think it might be the puppet-modules that should deploy that package on the client. https://bugzilla.redhat.com/show_bug.cgi?id=1049895
13:24:03 <ndipanov> there is an upstream process for this
13:24:10 <ndipanov> creating a milestone branch etc
13:24:11 <giulivo> ndipanov, yeah so current RDO is based on 5th of Dec tarballs right?
13:24:27 <ndipanov> on 2014.1.b1 tarballs
13:24:38 <ndipanov> check out that tag in git and see what's there
13:24:51 <giulivo> heh, thanks!
13:32:21 <pradeep> kashyap:  bare metal provisioning is expected to work on RDO? is it tested?
13:33:49 <morazi> ohochman, one sec.  I think there is a related bug that implies that we aren't expecting openstack-selinux to be required...
13:35:10 <morazi> ohochman, lemme ask someone (https://bugzilla.redhat.com/show_bug.cgi?id=1015625 is the one I'm talking about the seems to at least imply openstack-selinux should not be a hard dep)
13:35:22 <ohochman> morazi: if so.. what will be dealing with the AVCs ?
13:35:41 <ndipanov> kashyap, btw that isciinitator bug is legit
13:35:48 <ndipanov> will report it now
13:36:52 <Dafna> ndipanov: hi
13:37:29 <ndipanov> Dafna, ohai
13:38:07 <Dafna> ndipanov: :) can i ask a silly question?
13:38:23 <ndipanov> Dafna, allways
13:38:28 <ndipanov> there are no silly questions
13:38:46 <Dafna> ndipanov: you should say, how is that different from what you usually ask ;)
13:39:03 <ndipanov> I really shouldn't ;)
13:39:21 <Dafna> ndipanov: lol
13:39:39 <Dafna> ndipanov: should I be able to run rebuild on an instance booted from volume?
13:39:39 <morazi> ohochman, yea, that's a completely fair point.
13:40:03 <ndipanov> Dafna, not sure need to check
13:40:15 <ndipanov> btw what are you testing this on? rhel?
13:40:22 <Dafna> ndipanov: yes.
13:40:34 <ndipanov> Dafna, that's why you can boot from volume
13:40:41 <ohochman> morazi: OK, I've updated the bug - the AVC's will be found the compute side as well.
13:40:43 <ndipanov> Dafna, gimme a sec
13:40:43 <Dafna> ndipanov: what do you mean?
13:40:52 <ndipanov> it's borked on f20 afaict
13:41:06 <Dafna> ndipanov: can it be selinux?
13:41:16 <Dafna> ndipanov: or is it something else?
13:41:49 <ndipanov> nah smth else - a regression fixed in rhel but not in fedora
13:42:03 <morazi> ohochman, the foreman related AVCs or are you talking about different  new AVCs?
13:43:27 <ndipanov> Dafna, should work yeah
13:43:37 <ndipanov> Dafna, if it's not it's an upstream bug likely
13:43:49 <ohochman> morazi: well the thing is. that foreman do not deploy openstack-selinux-0.1.3-2.el6ost.noarch on the neutron-controller / neutron-compute machines as opposed to packstack
13:45:32 <Dafna> ndipanov: thanks. and when I asked for a password when rebuild from horizon, is that a password change for the rebuild image or is that approval that I am allowed to rebuild?
13:46:05 <ndipanov> Dafna, you sure you're not talking about rescue?
13:46:14 <Dafna> ndipanov: i'm sure :)
13:46:33 <ndipanov> then I'd have to check Dafna
13:46:41 <Dafna> ndipanov: unless horizon use rescue and call it rebuild...
13:46:48 <Dafna> ndipanov: thanks :) sorry...
13:46:56 <morazi> ohochman, ah, perhaps that differentiation was not clear to everyone.
13:47:02 <Dafna> yrabl: ping
13:47:26 <ndipanov> Dafna, that would also be a bug
13:47:34 <morazi> ohochman, so a packstack install always puts down openstack-selinux even though there isn't an rpm dep there?
13:47:36 <Dafna> ndipanov: yes :)
13:47:51 <ohochman> morazi: I just check neutron machine that was deployed with packstack it has openstack-selinux installed and no AVC's in /var/log/messages.
13:48:08 <Dafna> yrabl: this is cool... create an empty volume -> boot instance from it -> rebuild the instance from an image
13:48:14 <ohochman> morazi: I guess it does .
13:51:25 <morazi> bcrochet|bbl, do you happen to know the answer there off the top of your head w/r/t the behavior of foreman vs packstack in terms of the presence of openstack-selinux -- it likely impacts https://bugzilla.redhat.com/show_bug.cgi?id=1015625 at least slightly
13:55:00 <Dafna> giulivo: I think volumes are treated like images when we boot from one. I can boot two instances from the same volume
13:56:33 <yrabl> Dafna, cool
13:56:37 <ohochman> bcrochet|bbl: ping
13:57:18 <giulivo> Dafna, wait when booted from volume the volume should be marked as in use and not be usable to boot another instance
13:57:24 <giulivo> is it marked as "in-use" ?
13:57:25 <morazi> ohochman, actually I may be misparsing that bz at least slightly, so yea let us talk a bit and see what we come up to.  I may be that we just need to ensure a package is pulled in to all host groups.
13:57:58 <Dafna> giulivo: I know... wanna look?
13:58:12 <giulivo> I can try that too, just asking if it is marked as in-use or not
13:58:24 <giulivo> as you shouldn't be allowed from _any_ volume marked as in-use
13:59:02 <Dafna> giulivo: yes
13:59:19 <ohochman> morazi: I agree. I think the packstack should be there and if it blocks the user to perform any action that's a bug that need to be resolve within the openstack-selinux.
14:00:02 <giulivo> Dafna, uh that's bad ... the can you just attach a volume to an instance and, after it is marked in-use, see if you can use it also to boot a new server?
14:00:17 <giulivo> in that case, the "in-use" filtering is probably just not working
14:01:17 <Dafna> giulivo: actually, because of the bug that I found before where the volume seem to be stateless (as in data is not saved once I destroy the instance) I think they treat the volume like an image and are creating some sort of snapshot
14:03:13 <giulivo> Dafna, let me do some tests around that too, are you doing all this on guster?
14:04:20 <Dafna> giulivo: yep
14:06:10 <ayoung> ohochman, can't talk now, got a meeting in a minute.  I'll check in once I'm back
14:07:56 <Dafna> giulivo: I found what is wrong...
14:08:22 <Dafna> giulivo: do this... boot an instance from a volume -> rebuild the volume from an image (its no longer attached)
14:09:10 <giulivo> Dafna, rebuild the server you mean, right?
14:09:22 <Dafna> giulivo: so once we rebuild the volume, it's marked with an image and it seems that the active image is no longer the volume
14:09:27 <Dafna> giulivo: yes
14:09:36 <Dafna> giulivo: rebuild the server
14:10:03 <giulivo> Dafna, so that is correct, when you rebuild it basically "frees" the volume
14:10:10 <rlandy> how long should it take to delete a F19 instance. looks to be hanging in 'Deleting' state
14:10:24 <giulivo> Dafna, and it can be used by other instances
14:10:33 <giulivo> but the data in it should be _persistent_
14:10:39 <giulivo> that's one more thing I want to check
14:10:50 <giulivo> no way you should lose data from volumes
14:10:58 <Dafna> giulivo: so I think that the volume is detached and the vm is started again from an image
14:11:04 <Dafna> giulivo: not sure its a bug...
14:11:15 <giulivo> Dafna, it's the correct behaviour if you rebuild from image
14:11:29 <Dafna> giulivo: not related to the data bug - that happens 100% no rebuild or anything
14:11:33 <Dafna> giulivo: yes
14:11:48 <giulivo> yeah how do I reproduce that? just attach write some data , detach and the attach somewhere else?
14:11:58 <giulivo> for my own curiosity
14:12:04 <bcrochet> ohochman: pong
14:12:24 <giulivo> as cinder should touch the data at all so I wonder why it's not there afterwards and what is actually attached to the second instance
14:12:33 <giulivo> *shoult NOT touch
14:12:34 <Dafna> giulivo: boot an instance from volume -> write data -> destroy instance -> boot a second instance from the same volume -> check if the data is there
14:12:43 <giulivo> Dafna, I see, will try!
14:12:46 <giulivo> thanks!
14:12:51 <Dafna> giulivo: no prob
14:13:42 <kashyap> shardy, Thanks (late response)
14:14:03 <ohochman> bcrochet: I wondered about the "Med" in the neutron 4 nodes scenario ? http://openstack.redhat.com/TestedSetups_2014_01#Advanced_Installs_.28Foreman_Based.29_--_Work_in_Progress
14:14:03 <kashyap> ndipanov, You have a URL about the iscsi bz?
14:14:31 <ndipanov> kashyap, https://bugzilla.redhat.com/show_bug.cgi?id=1049922
14:14:41 <kashyap> pradeep, I haven't tested bare metal myself. If you have spare cycles, please try it & let us update on the list.
14:16:25 <larsks> kashyap: Did anyone else try out icehouse on F20 yesterday that you know of?  I ran into all sorts of exciting neutron/selinux issues and curious if I was just lucky or if this is an actual thing.
14:17:17 <pradeep> pradeep: i have RDO running fine with KVM nodes. But bare metal provisioning, i feel not at all stable.
14:17:21 <pradeep> kashyap: ^
14:17:22 <kashyap> larsks, /me is on Icehouse F20. All the bugs I'm filing are related to that.
14:17:56 <larsks> kashyap: Are you running with selinux enabled?  I ended up needing at least this to get neutron running: https://gist.github.com/larsks/8306690
14:17:56 <pradeep> kashyap: community had plans to move bare metal away from nova.  probably it might be stable later.
14:18:03 <kashyap> pradeep, "not all stable" is  very vague, if you can, please post detailed observations to rdo-list@redhat.com
14:18:07 <pradeep> kashyap: any way, i am trying now. will let you
14:18:15 <kashyap> larsks, Yes, that was my next point (SELinux)
14:18:21 <kashyap> larsks, This is what you need :-)  -- https://bugzilla.redhat.com/show_bug.cgi?id=1049817
14:18:30 <pradeep> pradeep:  have been updating/sending mails to community. i dint get any responce :)
14:18:34 <pradeep> kashyap: ^
14:18:42 <pradeep> kashyap:  let me try again. :)
14:18:57 <kashyap> pradeep, Patience is your friend.
14:19:11 <pradeep> kashyap:  :) hehe excellent.
14:19:12 <kashyap> nmagnezi, There you go, SELinux issues are fixed in this -- selinux-policy-3.12.1-113.fc20
14:19:17 <larsks> kashyap:  Thanks.
14:19:50 <larsks> kashyap: Sort of wish selinux policies weren't so monolithic (so that we could package the appropriate module with neutron).  Ah well.
14:20:18 <kashyap> larsks, Indeed. That was mail in drafts! I had to follow up & bug SELinux maintainers to get this fixes in
14:20:59 <Dafna> yrabl: did you test boot from volume (create new volume)>
14:21:25 <yrabl> Dafna, not yet/
14:21:37 <kashyap> #action Kashyap/larsks to bring up the issue of modularized SELinux policies for OpenStack on the list
14:21:46 <Dafna> yrabl: :) ok... we have bugs there. several
14:21:51 <nmagnezi> kashyap, how can I use this in RHEL?
14:22:55 <kashyap> nmagnezi, I assumed you also had some env. based on F20 (no worries). I don't think mgrepl backported these fixes to EPEL.
14:23:07 <bcrochet> ohochman: It just signifies that I have some tempest failures.
14:23:17 <kashyap> (He was kind enough to quickly make this build despite a deluge of other issues he was fixing)
14:24:05 <nmagnezi> kashyap, any chance he will port this to epel for rhel?
14:27:53 <kashyap> nmagnezi, I doubt he has time to do it today.
14:28:22 <kashyap> I posted a message, waiting for his response
14:28:50 <morazi> ohochman, ok, thx for filing the bug.  I've asked bcrochet to start looking at it.  Shouldn't be hard as much as requires confirmation that it doesn't break things somewhere else.
14:36:36 <ohochman> bcrochet: morazi: I've manage to set that setup neutron-controller + neutron-networker + 2X neutron-compute. and I booted 7 instances on it .
14:37:09 <ohochman> bcrochet: morazi : If you want we can run tempest on that setup as well.
14:37:35 <bcrochet> ohochman: morazi That would be ideal I would think, to run tempest.
14:51:37 <mbourvin> any special reason why I can't create a new volume from an existing volume from horizon?
14:53:53 <yrabl> Dafna, I'm not able to try that at the moment - the keystone isn't working properly :)
14:54:37 <Dafna> yrabl: I'm doing it. what do you mean keystone isnt working properly?
14:55:42 <yrabl> when I sent a command it got stuck for like 20-30 seconds on keystone alone
14:55:45 <yrabl> Dafna,
14:55:56 <yrabl> Dafna, I'm checking it at the moment
14:56:08 <Dafna> yrabl: who's helping you?
14:57:48 <yrabl> Dafna, no one, right now, I'm trying to understand what is happening:1. the keystone is very slow. 2. the CC fail to delete an instance
14:58:50 <Dafna> yrabl: ajeain1 can you please help yogev with the issue he found for keystone?
15:00:22 <yrabl> Dafna, it was a problem with Neutron
15:00:56 <Dafna> yrabl: call me
15:31:10 <giulivo> Dafna, did you open a bug for the volumes data loss?
15:31:29 <Dafna> giulivo: yes
15:31:39 <Dafna> giulivo: did it happen for you?
15:31:44 <giulivo> can you add me on CC there? I just reproduced yes
15:31:55 <Dafna> giulivo: sure... I have to find it :)
15:41:15 <Dafna> giulivo: you would want to see this too https://bugzilla.redhat.com/show_bug.cgi?id=1049995
15:41:25 <Dafna> giulivo: I liked this bug...
15:42:12 <giulivo> doh
15:43:24 <mpavlase> uff RDO si quite hungry... VM with 2GB of memory isn't enough for it...
15:46:26 <larsks> mpavlase: 2GB for a compute node should work, as long as you're only starting a single m1.tiny or smaller.  I use 2048 for my compute and 2300 MB for my controller.
15:47:28 <mpavlase> larsks: neutron used a lot of memory
15:48:01 <larsks> My neutron controller has 1GB and seems happy.
15:48:22 <larsks> I'm only creating a single network and booting a Cirros instance for testing, though.
15:49:41 <mpavlase> larsks: hmm... my VM (as host) wasn't able to start mysqld service due to lack of memory
15:50:42 <larsks> Note that I'm running neutron on a separate host from the other controller services.
15:50:48 <mpavlase> larsks: so it will not me allow to do almost nothing in openstack
15:50:58 <mpavlase> larsks: that would be it
15:51:20 <larsks> That's what I meant when I said
15:51:25 <larsks> ...ooops...
15:51:38 <larsks> "my neutron controller has 1GB" <-- separate from other services
15:51:43 <ndipanov> kashyap, you there?
15:52:17 <kashyap> ndipanov, Yes, severely multi-tasking (should reduce it!)
15:52:49 <ndipanov> kashyap, so when you come around to this - I am getting the ECONNREFUSED only from neutron services
15:53:05 <ndipanov> you had that some time ago... remember what the solution was?
15:53:13 <ndipanov> if I remember correctly
15:54:28 <larsks> ndipanov: if you've confirmed that the services are running, have you checked for appropriate firewall rules? Port 9696 needs to be open.
15:55:20 <ndipanov> larsks, was about to - but I just restarted the node - no reason why it shoud change...
15:56:06 <larsks> Mmmmmaybe.  Are you running Fedora?
15:56:19 <ndipanov> larsks, yes
15:56:26 <larsks> Do you have firewalld installed?
15:56:31 <larsks> (...and running...)
15:56:32 <ndipanov> of course
15:56:35 <kashyap> ndipanov, I lost track of what resolved it, I've been chasing SELinux issues, 1 sec.
15:56:46 <ndipanov> kashyap, thanks no rush
15:57:20 <larsks> Because my recollection is that packstack doesn'tk now about firewalld, so it doesn't do the right thing to make persistent firewall rules.
15:57:33 <larsks> My F20 system is currently installed right now, so I can't check for certain.
15:57:48 <larsks> I could be wrong.
15:59:00 <kashyap> pixelb, Thanks for the typo spotting, that was meant to be still selinux-policy-3.12.1-113.fc20.
15:59:30 <kashyap> mgrepl mentioned that's for testing only, and he plans to re-spin & submit an update to Bodhi again today with some more issues he's fixing
16:01:47 <giulivo> Dafna, so I couldn't reproduce that consistently, but I had it with thinlvm ... seems something related to buffering of data not synced
16:01:57 <giulivo> (well, not synced on time)
16:02:03 <ndipanov> larsks, yeah it's that most likely - the port is disabled
16:02:26 <Dafna> giulivo: reproduce what? the create vol issue?
16:02:36 <giulivo> Dafna, the "data loss"
16:02:55 <Dafna> giulivo: It reproduced for me 100% :)
16:03:11 <pixelb> kashyap, thanks. I hope https://bugzilla.redhat.com/1048946 is fixed by that too
16:05:21 <Dafna> giulivo: when we rebuild a server from horizon we are asked for a password. is that for setting the password for the server we are building or is it authentication for the user running this action?
16:06:26 <giulivo> Dafna, it's the server password
16:07:05 <Dafna> giulivo: :) thanks
16:07:34 <giulivo> it should be hijacking that by loopmounting the dick
16:07:37 <giulivo> disk
16:07:40 <giulivo> :(
16:07:52 <Dafna> giulivo: ?
16:08:27 <giulivo> the server password I mean, nova should be injecting it by mounting the disk image
16:08:44 <Dafna> giulivo: ah... and its not what the api will call rescue?
16:09:17 <ndipanov> kashyap, hey what happened to that db bug you had
16:09:20 <Dafna> giulivo: I want to understand if we actually rebuilding the image or if we are changing root pass
16:09:30 <ndipanov> I think it's not a real bug - I've not seen it yet
16:09:43 <kashyap> pixelb, Yes, it will be, that's the AVC, right - allow glance_api_t amqp_port_t:tcp_socket name_connect;
16:10:08 <pixelb> y
16:10:10 <kashyap> ndipanov, Hey, I still have that environment intact, just have to find 15-20 quiet minutes to do proper investigation & make a report
16:10:24 <ndipanov> k sure no rush
16:11:40 <giulivo> Dafna, no rebuild and rescue are two different things, rebuild should be rebooting the instance with a different image
16:13:44 <ndipanov> larsks, yeah firewalld was messed up
16:13:54 <Dafna> giulivo: good, so why would horizon ask for password? I mean... lets say I am a user who does not know what the root password to the image is (perhaps on purpose) if I have permissions to rebuild an instance I can have access to an image which I was not given permissions for...
16:13:58 <ndipanov> pixelb, are we reporting that against packstack or?
16:14:41 <Dafna> giulivo: what do we ask if the image was a windows os? do we know how to change Administrator password?
16:14:47 <larsks> puppet module and package question: is there any work out there to make puppet modules know about firewalld?  Also, what about firewalld service definitions in the various packages?
16:14:57 <larsks> ...well, that was meant for a different window.
16:15:02 <giulivo> Dafna, you can set the root password also when creating a new image so I think the idea is, if you have permissions to launch instances (and inject a password) you should be able to do it when rebuild too
16:15:04 <pixelb> ndipanov, I was reporting AVCs against selinux-policy or are you talking about something else?
16:15:25 <ndipanov> pixelb, I think smth else
16:15:38 <ndipanov> when I reboot hosts - firewall rules get messed up
16:15:41 <ndipanov> on f20
16:15:48 <ndipanov> I assume due to firewalld
16:15:52 <pixelb> ah ok
16:15:59 <ndipanov> pixelb, I'd say it's a bug
16:16:21 <pixelb> I vaguely remember that being already reported at least. checking..
16:16:27 <ndipanov> pixelb, thanks
16:16:41 <ndipanov> also did you see my bug against iscsiadmin-initiator?
16:16:59 <giulivo> Dafna, no that won't work for windows... this is actually called password injection and I think it is planned to be removed completely as one is expected to set the root password (if needed) by passing metadata to cloud-utils at launch time ... ndipanov, makes sense?
16:17:27 <pixelb> ndipanov, larsks logged this last Nov: https://bugzilla.redhat.com/1029929
16:18:00 <larsks> ...which is CLOSED DUPLICATE of https://bugzilla.redhat.com/show_bug.cgi?id=981583'
16:18:21 <Dafna> giulivo: so it's not only not removed but part of the rebuild process through horizon...
16:18:26 <larsks> ...which smells really really stale.
16:18:29 <pixelb> There was also looking from the other side: https://bugzilla.redhat.com/981652
16:18:31 <ndipanov> giulivo, without knowing exactly what you are talking about - I'd say yes - injection should go away I think - not sure if there are patches already but there was talk about ut
16:19:16 <giulivo> thanks ndipanov for helping there
16:19:53 <giulivo> Dafna, currently it's there yes, you can also set the password when creating a new instance though
16:19:54 <ndipanov> pixelb, thanks - so won't spam bugzilla more
16:20:37 <larsks> Well, you may want to comment on one of those bugs indicating that you're still seeing the problem with F20 and recent RDO packages.
16:20:58 <Dafna> ndipanov: remember when I asked about rebuild? so doing server rebuild through horizon will ask for a new root password on the instance you are rebuilding. do you know when will it go away?
16:21:28 <ndipanov> Dafna, yeah no idea really bit let me try it now that I have the setup working
16:21:32 <kashyap> ndipanov, If you're still around for a couple of mins, I can start taking a look at it now
16:21:32 <ndipanov> but*
16:22:08 <Dafna> ndipanov: thanks :)
16:22:20 <TVR_> I am sorry to ask... but anyone here an expert or knowledgeable with ceph as a backend for RDO openstack?
16:22:36 <ndipanov> kashyap, yeah I'll be here till 19:00 or something
16:22:58 <ndipanov> kashyap, I think we're on the same time these days :)
16:24:06 <kashyap> ndipanov, Yeah, I'll leave in ~10 days though (visa expiration)
16:25:40 <ndipanov> we'll if it's any consolation, I won't make it to fosdem so you won't miss me there :)
16:26:27 <kashyap> :-) LOL. We'll get some productive work done instead.
16:27:38 <kashyap> I was at FOSDEM 2012,  I presented on PKI, etc & couldn't attend many of the 5000 sessions in the 2 days :-). Whole 2 days I spent in the virt-dev room & met some folks
16:32:17 <pixelb> larsks, ndipanov I bumped up priority of 981583 FWIW
16:32:40 <larsks> pixelb: Writing an email about that right now :)
16:33:54 <ndipanov> pixelb, yeah - should really see some love - maybe move it to 20 as well
16:34:07 <ndipanov> pixelb, also https://bugzilla.redhat.com/show_bug.cgi?id=1049922
16:35:04 <ndipanov> pixelb, from what I can see in iscsi-initiator dist git - this is solved for rhel6.5 but will likely bite us for rhel 7 so good to solve it
16:35:52 <pixelb> ndipanov, ugh only rhel was patched :( was the fix sent upstream? Fedora should be patched in any case
16:37:03 <ndipanov> pixelb, not sure tbh - seems like a regression from a backport in rhel
16:37:10 <ndipanov> but keeps happening on fedora
16:37:23 <ndipanov> pixelb, but i didn't dig too deep
16:38:03 <ndipanov> btw 2 node cluster on my poor laptop working nicely now on f20 - just no volumes due to that :)
16:39:01 <pixelb> ndipanov, cool! make sure to update tested setups on the RDO wiki
16:39:19 <ndipanov> I did I think but still marked it as fail
16:58:19 <kashyap> ndipanov, Heya, so, the root-cause still is - first my compute-service is not up & running due to that libvirt 'avx' issue
16:58:33 <kashyap> I'm updating to openstack-nova-2014.1-0.5.b1.fc21 & trying
16:59:26 <ndipanov> kashyap, cool
17:01:32 <kashyap> Yay, it comes up. Now off to find out what's up with that nova instance hung in "deleting" state
17:03:05 <psedlak> anyone seen "SequenceError: execute() got an unexpected keyword argument 'maskList'" raised from /usr/lib/python2.6/site-packages/packstack/installer/core/sequences.py ?
17:07:56 <ndipanov> kashyap, I also noticed that I lose routing information on my hosts when I restart them and hence can't seem to ping instances
17:08:04 <ndipanov> kashyap, although that might not be the only reason
17:08:18 <ndipanov> kashyap, have you seen that before
17:09:05 <kashyap> ndipanov, Host as in Controller/Compute hosts? I didn't see it on my Havana F20 setup at-least
17:09:14 <ndipanov> yeah
17:09:34 <ndipanov> ip route gives me only my networks
17:09:42 <ndipanov> not the public one
17:09:44 <kashyap> Ice-house: I've been resolving issues one by one. Now pkg update on my Controller node is in progress; will let you know if I can get to it
17:09:54 <kashyap> ndipanov, GRE + OVS?
17:10:11 <ndipanov> the basic allinone so vlans
17:11:00 <kashyap> Ok: I don't have a VLAN setup, so - I cannot definitively answer. Let's invoke the spirit of larsks
17:14:14 <larsks> The what now?
17:14:35 <ndipanov> larsks, you are a neutron guy?
17:14:53 <larsks> I play one on TV.  Depends on your issues :)
17:15:16 <larsks> Reading through scrollback...
17:15:32 <kashyap> ndipanov, So, I see something strange: compute service is "active" for a brief second or so, then it goes into inactive (dead)
17:15:50 <ndipanov> larsks, thanks
17:15:56 <ndipanov> kashyap, log?
17:16:04 <kashyap> Up-coming. . .
17:16:21 <kashyap> It's still the libvirt error; but there must be something else too
17:16:49 <larsks> ndipanov: I'm not sure I understand your issue exactly.  What routes are disappearing?
17:17:43 <ndipanov> larsks, so on my "controler" node
17:17:43 <kashyap> ndipanov, That' the Compute log - http://paste.fedoraproject.org/66759/01418138/
17:17:51 <ndipanov> where all the neutron services are running
17:18:06 <ndipanov> I don't have a route to the "public" network
17:18:28 <ndipanov> larsks,
17:18:29 <ndipanov> ^
17:19:23 <larsks> Is your public network associated with an actual interface on your controller?
17:20:33 <larsks> Can you post "neutron net-list", "neutron subnet-list", "neutron router-list", and "ip route" and "ip addr show" somewhere?
17:20:54 <ndipanov> larsks, sure
17:21:08 <ndipanov> but I'd say it's not
17:24:21 <ndipanov> larsks, http://paste.openstack.org/show/60814/
17:24:35 <larsks> Looking...
17:25:15 <ndipanov> larsks, but iiuc that thing should be taken care of by the l3 agent
17:25:31 <larsks> What does subnet a513992a-9795-4325-ba4a-93cdf3da45d5 look like? It doesn't show up in the output of subnet-list.
17:25:58 <larsks> Also, can you add "ovs-vsctl show"?
17:26:18 <ndipanov> yeah sorry
17:27:18 <ndipanov> larsks, http://paste.openstack.org/show/60815/
17:31:07 <ndipanov> larsks, also dhcp not working for my instances...
17:31:47 <larsks> :/.  So, regarding your external network, do you recall what the route to it looked like before it disappeared?
17:31:49 <kashyap> pixelb, Isn't this a blocker? Compute service just dies after briefly coming up -- https://bugzilla.redhat.com/show_bug.cgi?id=1049391
17:32:20 <ndipanov> larsks, no, but it was there for sure as I was pinging and sshing like a boss
17:32:39 <ndipanov> larsks, am I right to assume that l3-agent service is supposed to configure this?
17:32:47 <ndipanov> larsks, I can dig from there?
17:33:21 <pixelb> kashyap, yep, that's a bad one, but have you tried the workaround on the wiki?
17:34:24 <larsks> ndipanov: I don't think the l3-agent is supposed to set this up.  Generally, one would have an interface part of br-ex in the host network context that would provide the necessary connectivity (or an address on br-ex itself).
17:34:27 <kashyap> pixelb, Ah, that works? I know afazekas pointed me to it; let me quickly try
17:35:12 <pixelb> kashyap, it might only stop compute from dying rather than actually fixing the issue
17:35:23 <kashyap> pixelb, Should I clone it to launchpad? Or you know if it's already there
17:35:40 <pixelb> I've not looked upstream, so I would clone to lp
17:35:52 <larsks> ndipanov: I'd like to try to replicate your setup.  Did you just run "packstack --allinone", or did you specify additional options?
17:35:54 <kashyap> pixelb, Right; I wanted to debug another nova related issue, just at-least want to see the service up, let me try
17:36:23 <ndipanov> larsks, I ran --alinone
17:36:30 <ndipanov> and it worked fine
17:36:43 <ndipanov> then I rebooted the node and added another one
17:37:02 <larsks> You mean you added a compute node?
17:37:04 <ndipanov> and re-ran packstack with the old answer file, but chnged networks
17:37:08 <ndipanov> yes
17:37:19 <ndipanov> and also I added an interface to the first node
17:37:35 <larsks> Okay.  Just out of question, what is tenant_network_type in /etc/neutron/plugin.ini?
17:39:30 <ndipanov> local
17:39:34 <ndipanov> larsks, ^
17:40:27 <larsks> So for anything other than an all-in-one install, that *can't* be local -- it needs to be gre or vlan or vxlan.
17:40:46 <larsks> When you add a compute node to an allinone install, you need to update your tenant network type.
17:41:03 <ndipanov> larsks, makes sense
17:41:16 <ndipanov> larsks, just for my information - what does that setting do?
17:42:00 <larsks> That controls the type of networks used for instance network traffic.  When set to "gre" (or possibly "vxlan"), compute nodes are connected to your neutron controller via a tunnel. When set to "vlan", connectivity is via vlans.  Tunnels are generally easier to work with.
17:42:35 <larsks> Make sure that you also have enable_tunneling=True.
17:42:37 <ndipanov> larsks, which would create an additional bridge iiuc?
17:42:40 <larsks> See http://openstack.redhat.com/Using_GRE_tenant_networks for some docs.
17:42:55 <larsks> That would create a new bridge br-tun on your neutron controller and compute nodes.
17:43:26 <ndipanov> yeah makes sense larsks
17:43:32 <larsks> See e.g. this picture: http://blog.oddbit.com/assets/quantum-gre.svg
17:43:54 <ndipanov> larsks, yeah got it
17:45:10 <ndipanov> larsks, thanks
17:46:06 <larsks> ndipanov: I've started an allinone install.  Going to go grab some lunch.
17:46:15 <ndipanov> larsks, enjoy
18:10:02 <kashyap> pixelb, That workaround doesn't fix it -- the service still just does after momentarily coming up.
18:10:09 <kashyap> s/does/dies
18:22:36 <kashyap> Anyhow, a libvirt dev confirms that something is passing the XML with 'avx' enabled twice.
18:39:22 <larsks> Bummer, looks like ndipanov has taken off.
19:14:38 <ndipanov> larsks, still there?
19:32:05 <larsks> Yup.  Do you have an /etc/sysconfig/network-scripts/ifcfg-br-ex?
19:49:27 <ndipanov> larsks, sorry missed your mst
19:51:25 <ndipanov> larsks, yes I do
19:52:52 <larsks> That should set up br-ex with an address on your public network (which should also result in creating the appropriate network route).  If you just run "ifup br-ex" do things look like you expect?  I.e., does br-ex have an address and does the route exist?
19:54:27 <ndipanov> larsks, I'll try
19:54:46 <ndipanov> but after reviewing docs some more
19:54:55 <ndipanov> I think setting the route up by hand
19:55:25 <ndipanov> to route public ips to br-ex should work too
19:55:34 <larsks> You shouldn't need to set up the route by hand.  It should get created as a byproduct of setting the ip on br-ex.
19:55:51 <ndipanov> yes you are right
19:57:47 <larsks> I see that running "ifup br-ex" on my allinone, after rebooting, does not properly configure the interface.  I'm looking into why right now.
19:58:06 <ndipanov> larsks, ifdown and ifup worked for me
19:59:01 <larsks> That works, but something isn't working at boot.  There may be a race condition or something with OVS getting set up.
20:00:24 <ndipanov> larsks, race condition between bringing up the interfaces and OVS?
20:00:45 <ndipanov> larsks, is that what you meant?
20:01:07 <larsks> Yes, something like that.
20:01:25 <ndipanov> that;s not really an openstack bug tho :)
20:01:51 <ndipanov> but yeah network host should make sure that the interfaces are up
20:03:26 <larsks> Sure, but it's still a bug that results in a non-functioning openstack deployment if you reboot your system.
20:03:34 <larsks> That's no fun.
20:03:35 <ndipanov> which service creates that bridge? I assume l3 agent since it's the otherside of the namespace that des the routing
20:03:46 <zaitcev> I may have a related issue, or I may not. But basically I followed test day scenario with packstack, installed everything, but instance networking is dead for some reason.
20:04:01 <ndipanov> setenforce 0 zaitcev ?
20:04:07 <larsks> zaitcev: For an all-in-one install?  On RHEL/F20/something else?
20:04:14 <ndipanov> policy is totaly broken on fed20
20:04:37 <ndipanov> anyway larsks - should l3-agent make sure it's up maybe?
20:06:07 <larsks> I don't think so.  External connectivity is sort of the province of the local administrator.  Depending on your configuration, you may not want an address on br-ex -- maybe you're using a real router somewhere else and you just need another interface in the bridge.
20:06:19 <zaitcev> ndipanov, larsks: More specifically - http://paste.fedoraproject.org/66823/
20:06:45 <ndipanov> larsks, well in that case you will disable the l3 agent no?
20:07:30 <larsks> I don't think so, no.  The l3 agent will still be responsible for hooking up routers to this network.
20:07:46 <larsks> The l3 agent handles the openstack side of things, and you handle the "local environment" side of things.
20:07:57 <ndipanov> larsks, hmmm ok
20:08:18 <ndipanov> zaitcev, ip netns list?
20:08:30 * ndipanov thinks he knows neutron now... :)
20:08:41 <zaitcev> [root@guren ~(keystone_admin)]# ip netns list
20:08:41 <zaitcev> qdhcp-5e9e313e-f13c-4b7e-b522-668cde803bba
20:08:59 <larsks> zaitcev: Run "ip netns exec qdhcp-5e9e313e-f13c-4b7e-b522-668cde803bba ip addr"
20:09:12 <ndipanov> zaitcev, also - you won't be able to ping them on their private IPs
20:09:24 <ndipanov> that network is not routed form the node
20:09:25 <larsks> Well, you can, by running "ip netns exec ... ping ..."
20:09:35 <ndipanov> larsks, yeah ok
20:09:49 <zaitcev> Okay. How do I fix this?
20:10:06 <larsks> It's not clear there's anything to fix from what you've shown us.  WHat's not working, exactly?
20:10:25 <zaitcev> I want my instances to have network connectivity.
20:10:50 <ndipanov> zaitcev, do they get ips
20:10:52 <ndipanov> ?
20:11:08 <ndipanov> private ips I mean
20:11:23 <larsks> If you want them to have external connectivity ("external" meaning "outside of openstack"), you'll need to create an external network, create a router, add an interface on localnet to the router, set up a gateway for the router, etc...
20:11:50 <zaitcev> Nice. Some all-in-one this turned out to be.
20:12:00 <larsks> Did you install by running "packstack --allinone"?
20:12:15 <zaitcev> Actually, wait. No, I modified the answer file.
20:12:44 <zaitcev> It had CONFIG_NEUTRON_LB_TENANT_NETWORK_TYPE=local
20:13:05 <larsks> That's a no-op if you're using OVS (you'd want CONFIG_NEUTRON_OVS_TENANT_NETWORK_TYPE).
20:13:49 <zaitcev> sorry, CONFIG_NEUTRON_OVS_TENANT_NETWORK_TYPE=local
20:14:40 <larsks> Yeah, that's what is used by default for an all-in-one install (but won't work if you add additional nodes).
20:17:15 <larsks> The --allinone install creates networks named "public" and "private" as well as the necessary router to connect things.
20:21:22 <ndipanov> larsks, got networking working wooohooo
20:21:37 <larsks> \o/
20:40:52 <ndipanov> hey larsks
20:41:02 <ndipanov> shouldn't br-tun have an IP on both nodes?
20:41:39 <larsks> No, br-tun doesn't have an address.  It just bridges br-int to the GRE or VXLAN tunnel.
20:42:53 <ndipanov> larsks, even with two nodes
20:43:00 <ndipanov> I mean it has to be on the management network,, no?
20:44:18 <larsks> Not directly, no.  Br-tun has a GRE endpoint that connects to an ip address on your network controller.  Your compute node needs to have a route to that address, but (a) that address has nothing to do with br-tun, and (b) it doesn't even need to be a directly connected network.
20:44:36 <larsks> Starting up a GRE tunnel is no different from any other IP connection (e.g., ssh).
20:45:36 <larsks> So typically, a compute node will have an interface dedicated to instance traffic (we'll call it "eth2"), and that *interface* will have an address on a network dedicated to instance traffic (or maybe just the management network).
20:45:38 * ndipanov parsing
20:45:53 <ndipanov> larsks, OK
20:46:05 <larsks> ...but that interface is not itself connected to br-tun.  It's just used as the local endpoint for the GRE tunnel.
20:46:20 <larsks> Is that making sense?
20:46:32 <ndipanov> larsks, yeah
20:47:29 <ndipanov> GRE tunnelled frame is basically in an IP packet that gets routed like any other IP packer orriginatig from a process on the machine
20:47:33 <ndipanov> is that correct
20:47:47 <larsks> That's correct.
20:48:01 <ndipanov> makes sense
20:48:43 <ndipanov> and it figures out the to part bcs that's how the tunnels are set up
20:48:56 <ndipanov> and then the vlan tagging untagging is done by br-int
20:48:59 <ndipanov> right?
20:49:49 <larsks> ...maybe?  That is, it depends on what direction you're talking about.  br-tun translates between vlan tags and the tunnel id equivalents.  br-int applies vlan tags to traffic emitted by an instance, and removes vlan tags from traffic going to an instance.
20:50:40 <ndipanov> larsks, that is what I meant (not really but clear now :) )
20:51:17 <ndipanov> so I think that firewalld still messes with a bunch of stuff
20:51:46 <ndipanov> but will try it tomorrow again on a fresh set up
20:52:03 <larsks> If you start with the F20 cloud images, they do not have firewalld installed.
20:53:15 <ndipanov> but killing firewalld or disanling it should also work right?
20:54:58 <larsks> If you want the firewall rules to show up automatically at boot, you'll also need to install iptables-services and "systemctl enable iptables".
20:56:54 <larsks> Also, both stop and disable the firewalld service (because you don't want it coming back next time you reboot)
20:59:14 <ndipanov> larsks, yeah that's what I meant sorry - and enable iptables service
20:59:42 <ndipanov> k - I'm off thanks for the help!!
21:00:00 <larsks> No problem!
21:49:52 * kashyap is going to end the meetbot IRC meeting tomorrow morning (CET)  which he started yesterday morning. Folks in US/other geos might still be testing, etc.
01:22:27 <misterr> ? - in the case where the cinder-volumes VG is linked to a loopback device (all-in-one installation), and I now want to use a real 100GB LVM partition , do I simply create a new VG (e.g. - cinder-volumes-lvm), update the volume_group parameter in cinder.conf and restart?
05:46:06 <anand_> rdo 2 node installtion. how to make it highly available?
07:22:47 <kashyap> Morning; /me is going to end the meetbot IRC meeting now.
07:22:49 <kashyap> #endmeeting