- Users - oVirt List Archives

Re: Creating a static route
by Lev Veyde 20 Feb '19

20 Feb '19

Hi Doug, Thanks for the update, I'm glad that we could help and everything worked out just fine! I also opened a BZ, asking to add a note in the documentation regarding this issue. Thanks in advance, On Mon, Feb 18, 2019 at 7:15 PM Doug Maxfield <Doug.L.Maxfield(a)emcins.com> wrote: > Lev, > > Just wanted to update you on the progress. We created the persistent > static routes and rules and the data is routing correctly over the required > gateway. > > > > Thank you for all of your help!!! > > > > [image: Count On EMC] <http://www.emcins.com/> > > *Doug Maxfield *| Senior Operating Systems Analyst > > *EMC Insurance Companies *717 Mulberry St | Des Moines, IA 50265 > Tel: 515.345.4507 | Fax: 866.331.1522 > Doug.L.Maxfield(a)EMCIns.com | www.emcins.com > > > > > > *From:* Lev Veyde <lveyde(a)redhat.com> > *Sent:* Thursday, February 14, 2019 9:14 AM > *To:* Doug Maxfield <Doug.L.Maxfield(a)EMCIns.com> > *Cc:* users <users(a)ovirt.org> > *Subject:* Re: [ovirt-users] Creating a static route > > > > Hi Doug, > > Just one more thing that I forgot to mention, even though it may be > obvious - you can use subnets in both rules and routes. > > So if all the IPs you want to route sit in the same subnet then you can > contain them with a single command e.g. "to 172.22.20.0/24 priority 32764" > > Thanks in advance, > > > > On Thu, Feb 14, 2019 at 4:59 PM Lev Veyde <lveyde(a)redhat.com> wrote: > > Hi Doug, > > Sure, np. > > > > >> 1. - yes it is the priority setting of the specific rule. Basically you > put in the file the post "ip rule add" part of the command e.g. for "ip > rule add to 172.22.20.31 priority 32764" you need to put a line of "to > 172.22.20.31 priority 32764" > > >> 2. - yes, the same priority can be used for all of them. E.g. if you > want to route 172.22.20.31, 172.22.34.56 and 172.22.67.89 you will need to > add: > > to 172.22.20.31 priority 32764 > to 172.22.34.56 priority 32764 > to 172.22.67.89 priority 32764 > > into the rule-<interface name> file e.g. rule-ovirtmgmt or rule-eno3. > > > > Please note that you will also need to add the static route for each one > of these as well w/ "ip route add ... " command e.g. "ip route add > 172.22.20.31 via 172.21.12.61". > > You can also use the route-<interface name> putting just the route info > part, similar to the "ip rule" e.g. for equivalent of "ip route add > 172.22.20.31 via 172.21.12.61" you need put in the file a line with the > content "172.22.20.31 via 172.21.12.61". > > For routes it's important to use specifically the route-ovirtmgmt file > since the script automatically adds the device to the route command i.e. > you can put the following lines into your route-ovirtmgmt file: > > 172.22.20.31 via 172.21.12.61 > 172.22.34.56 via 172.21.12.61 > 172.22.67.89 via 172.21.12.68 > > (In case the first 2 of the mentioned IPs should be routed through the > 172.21.12.61 gateway, and the last one should be routed through the > 172.21.12.68 gateway). > > > > For the reference you can take a look at the > /etc/sysconfig/network-scripts/ifup-routes script. > It does the actual processing of both "route" and "rule" files for all > network interfaces on a system. > > > > Please feel free to contact if you have further questions. > > > > Thanks in advance, > > > > On Thu, Feb 14, 2019 at 3:39 PM Doug Maxfield <Doug.L.Maxfield(a)emcins.com> > wrote: > > Lev, > > Thanks for the response. I was just getting ready to contact you about > this. I do have 2 follow-up questions, if you don’t mind. > > > > 1. When we were talking about creating the route that would be enabled > again after a reboot, you said to modify “the > /etc/sysconfig/network-scripts/rule-<interface name> e.g. rule-ovirtmgmt or > rule-eno3 (just put in the file "to 172.22.20.31 priority 32764”. Is > priority 32764 for the ip rule command? > 2. If I have multiple IPs that I would need to create a static route, > would the priority be 32764 for all of them? > > > > Once again, thank you for all of your help with this!! > > > > [image: Count On EMC] <http://www.emcins.com/> > > *Doug Maxfield *| Senior Operating Systems Analyst > > *EMC Insurance Companies *717 Mulberry St | Des Moines, IA 50265 > Tel: 515.345.4507 | Fax: 866.331.1522 > Doug.L.Maxfield(a)EMCIns.com | www.emcins.com > > > > > > *From:* Lev Veyde <lveyde(a)redhat.com> > *Sent:* Thursday, February 14, 2019 3:19 AM > *To:* Doug Maxfield <Doug.L.Maxfield(a)EMCIns.com> > *Cc:* users <users(a)ovirt.org> > *Subject:* Re: [ovirt-users] Creating a static route > > > > Hi Doug, > > Sorry that it took a bit of time to get back to you. > Managed to catch today somebody from the oVirt/RHV network team to discuss > the issue. > > It seems that we don't currently support custom static routes on the > hosts, so the workaround that I suggested is the way to do it. > > Thanks in advance, > > > > On Tue, Feb 12, 2019 at 11:12 PM Doug Maxfield <Doug.L.Maxfield(a)emcins.com> > wrote: > > Will do. Thanks again for all your help!! Looking forward to a > confirmation of the fix or a different solution. Either way, I know I have > a working fix. > > > > [image: Count On EMC] <http://www.emcins.com/> > > *Doug Maxfield *| Senior Operating Systems Analyst > > *EMC Insurance Companies *717 Mulberry St | Des Moines, IA 50265 > Tel: 515.345.4507 | Fax: 866.331.1522 > Doug.L.Maxfield(a)EMCIns.com | www.emcins.com > > > > > > *From:* Lev Veyde <lveyde(a)redhat.com> > *Sent:* Tuesday, February 12, 2019 3:09 PM > *To:* Doug Maxfield <Doug.L.Maxfield(a)EMCIns.com> > *Cc:* users <users(a)ovirt.org> > *Subject:* Re: [ovirt-users] Creating a static route > > > > Hi Doug, > > Just a small note - the commands in the rc.local should be placed *before* > the very last line (the touch ... command). > > Thanks in advance, > > > > On Tue, Feb 12, 2019 at 10:56 PM Lev Veyde <lveyde(a)redhat.com> wrote: > > Hi Doug, > > Thanks, you're welcome! > > It's generally possible to add persistent rules by either using the legacy > /etc/rc.d/rc.local boot script (chmod +x /etc/rc.d/rc.local and put the > relevant commands at the end of the file, that way you can actually run any > commands) or by using the /etc/sysconfig/network-scripts/rule-<interface > name> e.g. rule-ovirtmgmt or rule-eno3 (just put in the file "to > 172.22.20.31 priority 32764" - without quotes), however of course this > needs to be tested to verify that it works. > > I forwarded the issue to the network team and hopefully they will update > soon. > > > > Thanks in advance, > > > > On Tue, Feb 12, 2019 at 10:17 PM Doug Maxfield <Doug.L.Maxfield(a)emcins.com> > wrote: > > Lev, > > You have no idea how helpful you have been!!!!!! I’ve been having this > issue for over 2 months and starting to cause major problems. > > > > Please let me know if you find a better solution. I’m going to wait to > hear back from you before implementing. > > > > If we go with this workaround, is there a way to add the ip rule so it’s > persistent? I know how to make the ip route add persistent. > > > > [image: Count On EMC] <http://www.emcins.com/> > > *Doug Maxfield *| Senior Operating Systems Analyst > > *EMC Insurance Companies *717 Mulberry St | Des Moines, IA 50265 > Tel: 515.345.4507 | Fax: 866.331.1522 > Doug.L.Maxfield(a)EMCIns.com | www.emcins.com > > > > > > *From:* Lev Veyde <lveyde(a)redhat.com> > *Sent:* Tuesday, February 12, 2019 2:14 PM > *To:* Doug Maxfield <Doug.L.Maxfield(a)EMCIns.com> > *Cc:* users <users(a)ovirt.org> > *Subject:* Re: [ovirt-users] Creating a static route > > > > Hi Doug, > > Yes, it looks like it did. > > I'll try to catch somebody from my local oVirt/RHV networking team to take > a look at this issue, maybe there is a better solution for that issue than > my workaround. > > > > But meanwhile you can use this workaround (ip route add + ip rule add), > just please be aware that these should be run after each host reboot, since > these commands are not persistent. > > > > Thanks in advance, > > > > On Tue, Feb 12, 2019 at 10:07 PM Doug Maxfield <Doug.L.Maxfield(a)emcins.com> > wrote: > > Lev, > > Here’s the results. It appears to fix the issue!! > > > > From non-working system: > > > > -bash-4.2# ip rule add to 172.22.20.31 priority 32764 > > > > -bash-4.2# traceroute 172.22.20.31 > > traceroute to 172.22.20.31 (172.22.20.31), 30 hops max, 60 byte packets > > 1 CenteraAmesAN1.emcins.com (172.21.12.61) 0.487 ms 1.082 ms 1.313 ms > > 2 192.168.90.9 (192.168.90.9) 1.507 ms 2.090 ms 2.294 ms > > 3 pdputopcomm01.emcins.com (172.22.20.31) 1.403 ms 1.401 ms 1.348 ms > > > > [image: Count On EMC] <http://www.emcins.com/> > > *Doug Maxfield *| Senior Operating Systems Analyst > > *EMC Insurance Companies *717 Mulberry St | Des Moines, IA 50265 > Tel: 515.345.4507 | Fax: 866.331.1522 > Doug.L.Maxfield(a)EMCIns.com | www.emcins.com > > > > > > *From:* Lev Veyde <lveyde(a)redhat.com> > *Sent:* Tuesday, February 12, 2019 2:01 PM > *To:* Doug Maxfield <Doug.L.Maxfield(a)EMCIns.com> > *Cc:* users <users(a)ovirt.org> > *Subject:* Re: [ovirt-users] Creating a static route > > > > Hi Doug, > > OK, I think that I have a guess of what may have happened here... > > Can you please run the following command on the (non-working) host: > > ip rule add to 172.22.20.31 priority 32764 > > > > This needs to be run in addition to the ip route add ... command. > > > > And then please run the traceroute command and let me know if that fixed > the issue. > > > > Thanks in advance, > > > > On Tue, Feb 12, 2019 at 9:30 PM Doug Maxfield <Doug.L.Maxfield(a)emcins.com> > wrote: > > Lev, > > Here you go. > > > > From non-working system > > > > -bash-4.2# ip rule > > 0: from all lookup local > > 32764: from all to 172.21.0.0/16 iif ovirtmgmt lookup 2887058719 > > 32765: from 172.21.0.0/16 lookup 2887058719 > > 32766: from all lookup main > > 32767: from all lookup default > > > > > > From working system > > > > [root@paputopcomm04 ~]# ip rule > > 0: from all lookup local > > 32766: from all lookup main > > 32767: from all lookup default > > > > [image: Count On EMC] <http://www.emcins.com/> > > *Doug Maxfield *| Senior Operating Systems Analyst > > *EMC Insurance Companies *717 Mulberry St | Des Moines, IA 50265 > Tel: 515.345.4507 | Fax: 866.331.1522 > Doug.L.Maxfield(a)EMCIns.com | www.emcins.com > > > > > > *From:* Lev Veyde <lveyde(a)redhat.com> > *Sent:* Tuesday, February 12, 2019 12:35 PM > *To:* Doug Maxfield <Doug.L.Maxfield(a)EMCIns.com> > *Cc:* users <users(a)ovirt.org> > *Subject:* Re: [ovirt-users] Creating a static route > > > > Hi Doug, > > That shouldn't really matter, since ovirtmgmt is just a bridged interface, > and unless I'm missing something it shouldn't really affect routing. > > Can you please also provide the output of "ip rule" from both working and > non-working hosts ? > > > > Thanks in advance, > > > > On Tue, Feb 12, 2019 at 8:01 PM Doug Maxfield <Doug.L.Maxfield(a)emcins.com> > wrote: > > Lev, > > Here’s the ip route from the server that worked and didn’t work > > > > IP Route from server that worked > > > > [root@paputopcomm04 ~]# ip route > > default via 172.21.0.250 dev eno3 > > 169.254.0.0/16 dev eno3 scope link metric 1002 > > 169.254.0.0/16 dev eno4 scope link metric 1005 > > 172.17.0.0/16 dev eno4 proto kernel scope link src 172.17.53.18 > > 172.21.0.0/16 dev eno3 proto kernel scope link src 172.21.5.34 > > *172.22.20.31 via 172.21.12.61 dev eno3* > > 172.26.0.0/16 dev eno1 proto kernel scope link src 172.26.5.34 > > > > > > > > IP Route from server that didn't work > > > > -bash-4.2# ip route > > default via 172.21.0.250 dev ovirtmgmt > > 169.254.0.0/16 dev eno4 scope link metric 1004 > > 169.254.0.0/16 dev ovirtmgmt scope link metric 1021 > > 172.17.0.0/16 dev eno4 proto kernel scope link src 172.17.53.15 > > 172.21.0.0/16 dev ovirtmgmt proto kernel scope link src 172.21.5.31 > > *172.22.20.31 via 172.21.12.61 dev ovirtmgmt* > > 172.26.0.0/16 dev eno1 proto kernel scope link src 172.26.5.31 > > > > We have had our network team look and they don’t know. As I said, the was > setup by Commvault. It’s almost like the ovirtmgmt interface will not > allow a route add due to the way it’s configured? They, Commvault, are > trying different things within their software to get this working, but it’s > causing us more problems. I figured I would take a shot and see if > someone, who works with the oVirt software, would have any ideas. > > > > > > [image: Count On EMC] <http://www.emcins.com/> > > *Doug Maxfield *| Senior Operating Systems Analyst > > *EMC Insurance Companies *717 Mulberry St | Des Moines, IA 50265 > Tel: 515.345.4507 | Fax: 866.331.1522 > Doug.L.Maxfield(a)EMCIns.com | www.emcins.com > > > > > > *From:* Lev Veyde <lveyde(a)redhat.com> > *Sent:* Tuesday, February 12, 2019 11:51 AM > *To:* Doug Maxfield <Doug.L.Maxfield(a)EMCIns.com> > *Cc:* users <users(a)ovirt.org> > *Subject:* Re: [ovirt-users] Creating a static route > > > > Hi Doug, > > This is indeed quite weird... > > It looks like the OS for some reason disregards the static route that you > have added. > > BTW, have you verified that it's indeed was added by running "ip route" > following the "ip route add ..." command ? > > Maybe somebody from the network team has an idea of what may be the cause > of this issue... > > > > Thanks in advance, > > > > On Tue, Feb 12, 2019 at 7:36 PM Doug Maxfield <Doug.L.Maxfield(a)emcins.com> > wrote: > > Lev, > > I’m showing you the same outputs (ip a, ip route, traceroute, ip route > add, and traceroute) from another server in this group that doesn’t use the > ovirtmgmt for it’s default interface. > > > > [root@paputopcomm04 ~]# ip a > > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1 > > link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 > > inet 127.0.0.1/8 scope host lo > > valid_lft forever preferred_lft forever > > inet6 ::1/128 scope host > > valid_lft forever preferred_lft forever > > 2: eno3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen > 1000 > > link/ether 90:1b:0e:db:bc:c0 brd ff:ff:ff:ff:ff:ff > > inet 172.21.5.34/16 brd 172.21.255.255 scope global eno3 > > valid_lft forever preferred_lft forever > > inet6 fe80::921b:eff:fedb:bcc0/64 scope link > > valid_lft forever preferred_lft forever > > 3: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen > 1000 > > link/ether 90:1b:0e:e8:da:03 brd ff:ff:ff:ff:ff:ff > > inet 172.26.5.34/16 brd 172.26.255.255 scope global eno1 > > valid_lft forever preferred_lft forever > > inet6 fe80::921b:eff:fee8:da03/64 scope link > > valid_lft forever preferred_lft forever > > 4: eno2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN > qlen 1000 > > link/ether 90:1b:0e:e8:da:04 brd ff:ff:ff:ff:ff:ff > > 5: eno4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen > 1000 > > link/ether 90:1b:0e:db:bc:c1 brd ff:ff:ff:ff:ff:ff > > inet 172.17.53.18/16 brd 172.17.255.255 scope global eno4 > > valid_lft forever preferred_lft forever > > inet6 fe80::921b:eff:fedb:bcc1/64 scope link > > valid_lft forever preferred_lft forever > > [root@paputopcomm04 ~]# > > [root@paputopcomm04 ~]# > > [root@paputopcomm04 ~]# ip route > > default via 172.21.0.250 dev eno3 > > 169.254.0.0/16 dev eno3 scope link metric 1002 > > 169.254.0.0/16 dev eno4 scope link metric 1005 > > 172.17.0.0/16 dev eno4 proto kernel scope link src 172.17.53.18 > > 172.21.0.0/16 dev eno3 proto kernel scope link src 172.21.5.34 > > 172.26.0.0/16 dev eno1 proto kernel scope link src 172.26.5.34 > > [root@paputopcomm04 ~]# > > [root@paputopcomm04 ~]# > > [root@paputopcomm04 ~]# traceroute 172.22.20.31 > > traceroute to 172.22.20.31 (172.22.20.31), 30 hops max, 60 byte packets > > 1 ames-acc1-agg-1.emcins.com (172.21.0.246) 1.081 ms 1.816 ms 1.832 ms > > 2 192.168.200.30 (192.168.200.30) 0.639 ms 192.168.200.14 > (192.168.200.14) 1.063 ms 1.577 ms > > 3 192.168.200.101 (192.168.200.101) 3.110 ms ames-sw-wanrtr.emcins.com > (192.168.200.97) 3.191 ms 3.277 ms > > 4 192.168.100.26 (192.168.100.26) 4.146 ms 4.273 ms 4.394 ms > > 5 192.168.100.98 (192.168.100.98) 1.893 ms 2.364 ms 2.397 ms > > 6 192.168.100.50 (192.168.100.50) 5.707 ms 192.168.100.33 > (192.168.100.33) 5.134 ms 192.168.100.50 (192.168.100.50) 4.105 ms > > 7 pdputopcomm01.emcins.com (172.22.20.31) 1.359 ms 1.374 ms 1.351 ms > > [root@paputopcomm04 ~]# > > [root@paputopcomm04 ~]# > > [root@paputopcomm04 ~]# > > [root@paputopcomm04 ~]# > > [root@paputopcomm04 ~]# ip route add 172.22.20.31 via 172.21.12.61 > > [root@paputopcomm04 ~]# > > [root@paputopcomm04 ~]# > > [root@paputopcomm04 ~]# > > [root@paputopcomm04 ~]# traceroute 172.22.20.31 > > traceroute to 172.22.20.31 (172.22.20.31), 30 hops max, 60 byte packets > > 1 CenteraAmesAN1.emcins.com (172.21.12.61) 6.184 ms 6.475 ms 6.686 ms > > 2 192.168.90.9 (192.168.90.9) 1.770 ms 1.979 ms 2.189 ms > > 3 pdputopcomm01.emcins.com (172.22.20.31) 1.405 ms * 1.354 ms > > > > [image: Count On EMC] <http://www.emcins.com/> > > *Doug Maxfield *| Senior Operating Systems Analyst > > *EMC Insurance Companies *717 Mulberry St | Des Moines, IA 50265 > Tel: 515.345.4507 | Fax: 866.331.1522 > Doug.L.Maxfield(a)EMCIns.com | www.emcins.com > > > > > > *From:* Lev Veyde <lveyde(a)redhat.com> > *Sent:* Tuesday, February 12, 2019 11:07 AM > *To:* Doug Maxfield <Doug.L.Maxfield(a)EMCIns.com> > *Cc:* users <users(a)ovirt.org> > *Subject:* Re: [ovirt-users] Creating a static route > > > > Hi Doug, > > In most cases it should not be required, especially as my guess (again it > would be easier if you could post the output of the "ip a" and "ip route") > is that the eno1 is part of the ovirtmgmt bridge. > > What is the output of the "traceroute 172.22.20.31" that you get on the > host after you have added the static route? > > Thanks in advance, > > > > > > On Tue, Feb 12, 2019 at 6:58 PM Doug Maxfield <Doug.L.Maxfield(a)emcins.com> > wrote: > > Lev, > > One other question, since there are multiple enabled interfaces (ovirtmgmt > and eno1), do I need to specify an interface for the route add. > > > > [image: Count On EMC] <http://www.emcins.com/> > > *Doug Maxfield *| Senior Operating Systems Analyst > > *EMC Insurance Companies *717 Mulberry St | Des Moines, IA 50265 > Tel: 515.345.4507 | Fax: 866.331.1522 > Doug.L.Maxfield(a)EMCIns.com | www.emcins.com > > > > > > *From:* Lev Veyde <lveyde(a)redhat.com> > *Sent:* Tuesday, February 12, 2019 10:50 AM > *To:* Doug Maxfield <Doug.L.Maxfield(a)EMCIns.com> > *Cc:* users <users(a)ovirt.org> > *Subject:* Re: [ovirt-users] Creating a static route > > > > Hi Doug, > > What do you mean by "static route is refused"? > > What is the error message(s) you see? > > > > Can you please try to add the route with the following command: > > ip route add 172.22.20.31 via 172.21.12.61 > > > > If that doesn't work, then please provide more details, i.e. what is the > exact error message(s) you see, the version of your OS and oVirt, output of > "ip a" and "ip route" on your host(s), so that it will be easier to help > you. > > > > Thanks in advance, > > > > On Tue, Feb 12, 2019 at 6:43 PM Doug Maxfield <Doug.L.Maxfield(a)emcins.com> > wrote: > > Lev, > > Thanks for your response. > > > > If I attempt to manually add the route using route add: > > > > route add -net 172.22.20.31 netmask 255.255.255.255 gw 172.21.12.61 dev > ovirtmgmt > > > > This static route is refused and all traffic is routed over the default > gateway. > > > > But if I use a server that doesn’t use the ovirtmgmt interface and set the > same static route, the required data is routed correctly over that static > route. > > > > [image: Count On EMC] <http://www.emcins.com/> > > *Doug Maxfield *| Senior Operating Systems Analyst > > *EMC Insurance Companies *717 Mulberry St | Des Moines, IA 50265 > Tel: 515.345.4507 | Fax: 866.331.1522 > Doug.L.Maxfield(a)EMCIns.com | www.emcins.com > > > > > > *From:* Lev Veyde <lveyde(a)redhat.com> > *Sent:* Tuesday, February 12, 2019 10:34 AM > *To:* Doug Maxfield <Doug.L.Maxfield(a)EMCIns.com> > *Cc:* users <users(a)ovirt.org> > *Subject:* Re: [ovirt-users] Creating a static route > > > > Hi Doug, > > What is the exact problem you are having while attempting to add a static > route on the hosts? > > > > Thanks in advance, > > > > On Tue, Feb 12, 2019 at 5:23 PM <doug.l.maxfield(a)emcins.com> wrote: > > Good Morning, > New to oVirt. We are using this with a backup solution from Commvault. > The issue that we are having is that we need to setup a static route for > specific data between 2 remote sites. The ovirtmgmt is configured with the > correct IP and gateway for the server. We need to route data over a > different gateway so that we don't "max out" the default network connection > between the 2 sites. Example below > > Ovirtmgmt IP - 172.21.5.31 > Gateway IP - 172.21.0.250 > > We need any traffic with a destination of 172.22.20.31(Remote site) to > route over this gateway, 172.21.12.61 > > There are multiple servers in this configuration. Servers that are not > using the ovirtmgmt interface for their default, we are able to route the > data. The only problem is with attempting to setup a different static > route on the ovirtmgmt interfaces. > > Thanks in advance for any help!!! > > Doug > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/WGXSLHAP3X5OT… > > > > -- > > > > *Lev Veyde* > > Software Engineer, RHCE | RHCVA | MCITP > > Red Hat Israel > > <https://www.redhat.com> > > lev(a)redhat.com | lveyde(a)redhat.com > > <https://red.ht/sig> > > *TRIED. TESTED. TRUSTED.* <https://redhat.com/trusted> > > > NOTICE: This message (including any attachments) is intended for a > specific individual and may contain information that is either confidential > or legally protected. If you believe that it has been sent to you in > error, please reply to the sender that you have received the message in > error, then delete it. If you are not the intended recipient, you are > hereby notified that any retention, dissemination, distribution, or copying > of this communication is strictly prohibited. Thank you. EMC071856 > > > > -- > > > > *Lev Veyde* > > Software Engineer, RHCE | RHCVA | MCITP > > Red Hat Israel > > <https://www.redhat.com> > > lev(a)redhat.com | lveyde(a)redhat.com > > <https://red.ht/sig> > > *TRIED. TESTED. TRUSTED.* <https://redhat.com/trusted> > > > > -- > > > > *Lev Veyde* > > Software Engineer, RHCE | RHCVA | MCITP > > Red Hat Israel > > <https://www.redhat.com> > > lev(a)redhat.com | lveyde(a)redhat.com > > <https://red.ht/sig> > > *TRIED. TESTED. TRUSTED.* <https://redhat.com/trusted> > > > > -- > > > > *Lev Veyde* > > Software Engineer, RHCE | RHCVA | MCITP > > Red Hat Israel > > <https://www.redhat.com> > > lev(a)redhat.com | lveyde(a)redhat.com > > <https://red.ht/sig> > > *TRIED. TESTED. TRUSTED.* <https://redhat.com/trusted> > > > > -- > > > > *Lev Veyde* > > Software Engineer, RHCE | RHCVA | MCITP > > Red Hat Israel > > <https://www.redhat.com> > > lev(a)redhat.com | lveyde(a)redhat.com > > <https://red.ht/sig> > > *TRIED. TESTED. TRUSTED.* <https://redhat.com/trusted> > > > > -- > > > > *Lev Veyde* > > Software Engineer, RHCE | RHCVA | MCITP > > Red Hat Israel > > <https://www.redhat.com> > > lev(a)redhat.com | lveyde(a)redhat.com > > <https://red.ht/sig> > > *TRIED. TESTED. TRUSTED.* <https://redhat.com/trusted> > > > > -- > > > > *Lev Veyde* > > Software Engineer, RHCE | RHCVA | MCITP > > Red Hat Israel > > <https://www.redhat.com> > > lev(a)redhat.com | lveyde(a)redhat.com > > <https://red.ht/sig> > > *TRIED. TESTED. TRUSTED.* <https://redhat.com/trusted> > > > > -- > > > > *Lev Veyde* > > Software Engineer, RHCE | RHCVA | MCITP > > Red Hat Israel > > <https://www.redhat.com> > > lev(a)redhat.com | lveyde(a)redhat.com > > <https://red.ht/sig> > > *TRIED. TESTED. TRUSTED.* <https://redhat.com/trusted> > > > > -- > > > > *Lev Veyde* > > Software Engineer, RHCE | RHCVA | MCITP > > Red Hat Israel > > <https://www.redhat.com> > > lev(a)redhat.com | lveyde(a)redhat.com > > <https://red.ht/sig> > > *TRIED. TESTED. TRUSTED.* <https://redhat.com/trusted> > > > > -- > > > > *Lev Veyde* > > Software Engineer, RHCE | RHCVA | MCITP > > Red Hat Israel > > <https://www.redhat.com> > > lev(a)redhat.com | lveyde(a)redhat.com > > <https://red.ht/sig> > > *TRIED. TESTED. TRUSTED.* <https://redhat.com/trusted> > > > > -- > > > > *Lev Veyde* > > Software Engineer, RHCE | RHCVA | MCITP > > Red Hat Israel > > <https://www.redhat.com> > > lev(a)redhat.com | lveyde(a)redhat.com > > <https://red.ht/sig> > > *TRIED. TESTED. TRUSTED.* <https://redhat.com/trusted> > > > > -- > > > > *Lev Veyde* > > Software Engineer, RHCE | RHCVA | MCITP > > Red Hat Israel > > <https://www.redhat.com> > > lev(a)redhat.com | lveyde(a)redhat.com > > <https://red.ht/sig> > > *TRIED. TESTED. TRUSTED.* <https://redhat.com/trusted> > -- Lev Veyde Software Engineer, RHCE | RHCVA | MCITP Red Hat Israel <https://www.redhat.com> lev(a)redhat.com | lveyde(a)redhat.com <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>

3 4

Ovirt project source code download problem
by xilazz＠126.com 20 Feb '19

20 Feb '19

Hi,The dear. May I ask where I can find the source code of ovirt project? I want to test the environment through compilation and installation. At present, most of the methods are online installation, so I cannot modify some contents. Thank you very much for your help. Thank you very much.

3 3

How to upgrade ovirt node host?
by adam_xu＠adagene.com.cn 20 Feb '19

20 Feb '19

How, everyone. I have upgraded my engine from 4.2.8 to 4.3.0. but when I check upgrade for the node host, there's no updates available for upgrade. I checked the repo file in the host, two files available: /etc/yum.repos.d/ovirt-4.2.repo /etc/yum.repos.d/ovirt-4.2-dependencies.repo The documentation seems to be out of date. https://www.ovirt.org/documentation/upgrade-guide/chap-Updates_between_Mino… so How could I upgrade my node hosts to ovirt 4.3 ? yours Adam

5 6

VM stats are not shown in HE-web interface and MOM is not available
by v.voroncov＠scanru.ru 20 Feb '19

20 Feb '19

So the problem is that virtual machine statistics are not shown in Ovirt Virualization Manager and something is wrong with MOM service. [root@ovirt-04 ~]# service vdsmd status Redirecting to /bin/systemctl status vdsmd.service ● vdsmd.service - Virtual Desktop Server Manager Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2019-02-19 17:13:31 MSK; 16h ago Main PID: 17676 (vdsmd) Tasks: 63 CGroup: /system.slice/vdsmd.service ├─17676 /usr/bin/python2 /usr/share/vdsm/vdsmd ├─26249 /usr/libexec/ioprocess --read-pipe-fd 77 --write-pipe-fd 76 --max-threads 10 --max-queued-requests 10 └─26269 /usr/libexec/ioprocess --read-pipe-fd 82 --write-pipe-fd 81 --max-threads 10 --max-queued-requests 10 Feb 20 09:11:14 ovirt-04.scanru.local vdsm[17676]: WARN MOM not available. Feb 20 09:11:14 ovirt-04.scanru.local vdsm[17676]: WARN MOM not available, KSM stats will be missing. Feb 20 09:23:47 ovirt-04.scanru.local vdsm[17676]: WARN MOM not available. Feb 20 09:23:47 ovirt-04.scanru.local vdsm[17676]: WARN MOM not available, KSM stats will be missing. Feb 20 09:36:20 ovirt-04.scanru.local vdsm[17676]: WARN MOM not available. Feb 20 09:36:20 ovirt-04.scanru.local vdsm[17676]: WARN MOM not available, KSM stats will be missing. Feb 20 09:48:53 ovirt-04.scanru.local vdsm[17676]: WARN MOM not available. Feb 20 09:48:53 ovirt-04.scanru.local vdsm[17676]: WARN MOM not available, KSM stats will be missing. Feb 20 10:01:25 ovirt-04.scanru.local vdsm[17676]: WARN MOM not available. Feb 20 10:01:25 ovirt-04.scanru.local vdsm[17676]: WARN MOM not available, KSM stats will be missing.

2 1

ovirt 4.2.7.1 fails to deploy hosted engine on GlusterFS
by hunter86_bg＠yahoo.com 19 Feb '19

19 Feb '19

Hello Community, I'm trying to deploy a hosted engine on GlusterFS which fails with the following error: [ INFO ] TASK [Add glusterfs storage domain] [ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is "[Failed to fetch Gluster Volume List]". HTTP response code is 400. [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "deprecations": [{"msg": "The 'ovirt_storage_domains' module is being renamed 'ovirt_storage_domain'", "version": 2.8}], "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Failed to fetch Gluster Volume List]\". HTTP response code is 400."} I have deployed GlusterFS via the HyperConverged Option in Cockpit and the volumes are up and running. [root@ovirt1 ~]# gluster volume status engine Status of volume: engine Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick ovirt1:/gluster_bricks/engine/engine 49152 0 Y 26268 Brick ovirt2:/gluster_bricks/engine/engine 49152 0 Y 24116 Brick glarbiter:/gluster_bricks/engine/engi ne 49152 0 Y 23526 Self-heal Daemon on localhost N/A N/A Y 31229 Self-heal Daemon on ovirt2 N/A N/A Y 27097 Self-heal Daemon on glarbiter N/A N/A Y 25888 Task Status of Volume engine ------------------------------------------------------------------------------ There are no active volume tasks I'm using the following guide : https://ovirt.org/blog/2018/02/up-and-running-with-ovirt-4-2-and-gluster-st… And on step 4 - Storage - I have defined it as follows: Storage Type: Gluster Storage Connection: ovirt1.localdomain:/gluster_bricks/engine/ Mount Options: backup-volfile-servers=ovirt2.localdomain:glarbiter.localdomain Can someone hint me where is the problem ?

3 15

stucked snapshot, locked disk
by Jiří Sléžka 19 Feb '19

19 Feb '19

Hello, we are using ovirt 4.2.8.2-1.el7. One our user probably tried to preview taken snapshot but the task is stucked and never finished. Also disk is locked. Here is engine.log https://pastebin.com/izBJ1BUg I am not sure what went wrong except one java.lang.NullPointerException error... Could it be some bug? Also I would like to unlock disk and clean this task. I can see locked snapshot and disk [root@ovirt dbutils]# ./unlock_entity.sh -q -t snapshot -c Locked snapshots vm_id | snapshot_id --------------------------------------+-------------------------------------- df27ee13-9961-41d5-a4eb-bbb0aadf9086 | 58998020-08cb-4c84-8847-0cb6790724ba [root@ovirt dbutils]# ./unlock_entity.sh -q -t disk -c Locked disks vm_id | disk_id --------------------------------------+-------------------------------------- df27ee13-9961-41d5-a4eb-bbb0aadf9086 | f6405448-baf8-4cc2-8ee7-798ca11cad10 but no runnig tasks [root@ovirt dbutils]# ./taskcleaner.sh -o t but in ovirt-manager I see this stucked task... Preview VM Snapshot predsysprep of VM install_10_64bit_LTSB_UK_ucebny - Started: Feb 12, 2019, 11:28:04 AM Validating - Completed: Feb 12, 2019, 11:28:04 AM Executing - Completed: Feb 12, 2019, 11:28:15 AM Creating Volume - Completed: Feb 12, 2019, 11:28:15 AM Finalizing - Started: Feb 12, 2019, 11:28:15 AM what is best approach to unlock and clean this? Thanks in advance, Cheers, Jiri

6 7

ovirt node installation and deploy...
by matteo fedeli 19 Feb '19

19 Feb '19

Hi at all, during the deployment the system fail... http://oi63.tinypic.com/et9kd4.jpg During installation it's corretct select all disks? All disk must be without any partition table? My target is on the first disk ovirt node, second vmstore/engine volume and third for data. By watching this image, https://access.redhat.com/webassets/avalon/d/Red_Hat_Hyperconverged_Infrast… How can I set the LV size correctly?

2 6

Re: Migration of VMs across two Ovirt setups
by Luca 'remix_tj' Lorenzetto 19 Feb '19

19 Feb '19

Hello Dipak, i didn't used export domains, but directly data domain, which avoided me the long time of copying and importing back before powering on. Luca On Tue, Feb 19, 2019 at 8:59 AM Dipak D. Chaudhari <ddc.comp(a)gmail.com> wrote: > > > Thanks > > Migrated in same way. > > 1] Created new nfs export domain on Local data center > 2] Added export domain to local data center > 3] Powered off all VMs and exported all VM and related templates to export domain > 4] Enabled maintenance for domain and detached from Local Data Center Setup > 5] Added nfs domain created in step 1 to another Data Center > 6] Imported VMs with templates > 7] Enabled maintenance for domain and detached from Data Center > 8] Dismantled local data center > > Done :) > > > > > > On Mon, Feb 18, 2019 at 3:21 PM Luca 'remix_tj' Lorenzetto <lorenzetto.luca(a)gmail.com> wrote: >> >> Hello Dipak, >> >> we shut down all the vms in a storage domain, detached and then >> imported on the new environment. Once imported we moved the vms to a >> newer storage and the old one removed. >> >> I don't know what can happen with local storage, but i think works >> like shared storage. >> >> Luca >> >> On Sat, Feb 16, 2019 at 6:39 AM <ddc.comp(a)gmail.com> wrote: >> > >> > Hi, >> > >> > I have two ovirt setups having multiple hosts and cluster and around 100+ vms in one and around 32 in one setup. >> > >> > I want to dismantle one setup and before before doing so all vms are need to be migrated to one setup >> > >> > I tried with import vms from another setup using Libvert -- https://ovirt.org/develop/release-management/features/virt/KvmToOvirt.html but no luck connection not getting established. >> > >> > What if I simply remove one host from one setup having local storage(all vms are located on local storage) and simply add it to another setup? and then migrate VMs from local to shared storage and then dismantle host ! >> > >> > >> > Regards >> > Dipak Chaudhari >> > _______________________________________________ >> > Users mailing list -- users(a)ovirt.org >> > To unsubscribe send an email to users-leave(a)ovirt.org >> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >> > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/NWV54RQGZWPTA… >> >> >> >> -- >> "E' assurdo impiegare gli uomini di intelligenza eccellente per fare >> calcoli che potrebbero essere affidati a chiunque se si usassero delle >> macchine" >> Gottfried Wilhelm von Leibnitz, Filosofo e Matematico (1646-1716) >> >> "Internet è la più grande biblioteca del mondo. >> Ma il problema è che i libri sono tutti sparsi sul pavimento" >> John Allen Paulos, Matematico (1945-vivente) >> >> Luca 'remix_tj' Lorenzetto, http://www.remixtj.net , <lorenzetto.luca(a)gmail.com> > > > > -- > > Thanks & Regards, > Dipak D. Chaudhari > > -- "E' assurdo impiegare gli uomini di intelligenza eccellente per fare calcoli che potrebbero essere affidati a chiunque se si usassero delle macchine" Gottfried Wilhelm von Leibnitz, Filosofo e Matematico (1646-1716) "Internet è la più grande biblioteca del mondo. Ma il problema è che i libri sono tutti sparsi sul pavimento" John Allen Paulos, Matematico (1945-vivente) Luca 'remix_tj' Lorenzetto, http://www.remixtj.net , <lorenzetto.luca(a)gmail.com>

1 0

Cho thue xe 35 cho di cua khau Mong Cai - Quang Ninh
by anddyn810＠gmail.com 19 Feb '19

19 Feb '19

<div data-redactor="1">Vào những dịp nghỉ lễ hay đơn giản là dịp cuối tuần, nếu quý khách chưa tìm được một địa điểm nào thú vị để du lịch cùng gia đình và bạn bè thì tại sao không thử một lần khám phá những cửa khẩu quốc tế - nơi cửa ngõ quốc gia để trải nghiệm những cảm giác thú vị? Ngọc Minh đưa ra dịch vụ <b><i><a href="http://thuexeotongocminh.com/dia-chi-cho-thue-xe-35-cho-gia-re-uy-tin-tai-h…">Cho thuê xe 35 chỗ</a></i></b> đi cửa khẩu Móng Cái – Quảng Ninh nhằm phục vụ nhu cầu đó của khách hàng.</div> <br> <div data-redactor="1"><b><span style="color: #ff0000;" data-redactor="1">Liên hệ cho thuê xe 35 chỗ tham quan Cửa khẩu Móng Cái: 0983.22.9996 – 0967.27.9998</span></b></div> <br> <div data-redactor="1"><b>Ngọc Minh cho thuê xe 35 chỗ tham quan Cửa khẩu Móng Cái</b></div> <br> <div data-redactor="1"><i>Thông tin về lịch trình cho thuê xe 7 chỗ tham quan Cửa khẩu Móng Cái</i></div> <br> <div data-redactor="1">- Lịch trình: Đón tại Hà Nội – Cửa khẩu Móng Cái – Trả về Hà Nội</div> <br> <div data-redactor="1">- Số km dự kiến: 314km/ chiều</div> <br> <div data-redactor="1">- Cung đường đi: Qua QL5B – QL18</div> <br> <div data-redactor="1">- Thời gian di chuyển: 5 giờ 22 phút</div> <br> <div data-redactor="1"><i>Dòng xe cho thuê xe 35 chỗ tham quan Cửa khẩu Móng Cái</i></div> <br> <div data-redactor="1">Với nhiều năm hoạt động trong lĩnh vực du lịch vận tải, Ngọc Minh luôn gây được lòng tin với khách hàng bởi sự đảm bảo chất lượng xe và độ an toàn trong các chuyến đi. Quý khách hoàn toàn có thể lựa chọn dòng xe 35 chỗ tại công ty chúng tôi cho chuyến đi gia đình hay nhóm bạn của mình. Bởi dòng xe 35 chỗ tại Ngọc Minh đều là xe đời mới, nội thất đầy đủ, sang trọng, mang lại cảm giác thoải mái nhất cho khách hàng. Hiện tại chúng tôi đang có các loại xe 35 chỗ như sau:</div> <br> <div data-redactor="1">- Dòng xe 35 chỗ Isuzu Samco</div> <br> <div data-redactor="1">- Dòng xe 35 chỗ Thaco Town</div> <br> <div data-redactor="1"><img src="https://lh3.googleusercontent.com/-Qbi8Q4-NCME/XGuvaoeP5mI/AAAAAAAAFRs/YAhb…" alt="[IMG]" /></div> <div data-redactor="1">Ngọc Minh cho thuê xe 35 chỗ đi Móng Cái</div> <br> <div data-redactor="1"><i>Giá các dòng xe 35 chỗ cho thuê tham quan Cửa khẩu Móng Cái</i></div> <br> <div data-redactor="1">Đa số khách hàng đều cho rằng, xe đời mới thì mức giá cho thuê chắc cũng sẽ rất đắt. Nhưng nếu quý khách sử dụng dịch vụ thuê xe tại Ngọc Minh, chúng tôi cam kết sẽ mang tới mức giá hợp lý so với thị trường chung. Hiện tại giá của dòng xe 35 chỗ của công ty là 11.000đ/km.</div> <br> <div data-redactor="1">Tham khảo thêm: <b><i><a href="http://thuexeotongocminh.com/thue-xe-du-lich/thue-xe-45-cho/">Cho thuê xe 45 chỗ giá rẻ tại Hà Nội</a></i></b></div> <br> <div data-redactor="1"><b>Vài nét về Cửa khẩu Móng Cái – Quảng Ninh</b></div> <br> <div data-redactor="1">Cửa khẩu Móng Cái là một trong những cửa khẩu đầu tiên của Việt Nam. Nằm tại phường Hòa Lạc, thành phố Móng Cái, tỉnh Quảng Ninh, cửa khẩu này là nơi thông thương với cửa khẩu Đông Hưng, tỉnh Quảng Tây, Trung Quốc.</div> <br> <div data-redactor="1">Đây là một trong những điểm tham quan rất được ưa chuộng đối với du khách. Ghé thăm cửa khẩu Móng Cái, du khách chỉ cần đăng ký với bên biên phòng là có thể vào khu vực cửa khẩu để chụp ảnh cùng mốc 1369 cũng như vạch biên giới nằm trên cầu Hữu Nghị Bắc Luân.</div> <br> <div data-redactor="1">Ngoài ra, nếu muốn khám phá thành phố Đông Hưng, Trung Quốc, du khách hãy chuẩn bị sẵn 2 ảnh 4x6, CMND và giấy khai sinh nếu có trẻ em dưới 15 tuổi để làm sổ thông hành. Thủ tục khá nhanh gọn chỉ mất khoảng 1 tiếng đồng hồ là bạn có thể xuất cảnh qua cửa khẩu rồi.</div> <br> <div data-redactor="1">Và nếu đã đặt chân tới Móng Cái thì đừng quên ghé qua chợ Móng Cái để có thể mua được những món đồ vừa ý về làm quà cho gia đình và bạn bè nhé!</div> <br> <div data-redactor="1">Còn chần chừ gì nữa mà không liên hệ ngay với <a href="https://www.facebook.com/chothuexedulichngocminh/"><b><i>Cho thuê xe 35 chỗ Hà Nội</i></b> </a>để có một chuyến đi hoàn hảo, đảm bảo an toàn và chất lượng tuyệt đối? Hãy gọi vào hotline <b><i><span style="color: #ff0000;" data-redactor="1">0983.22.9996 – 0967.27.9998</span></i></b> để được tư vấn cụ thể về lịch trình cũng như cách thức thuê xe.</div> <br> <div data-redactor="1">Sự tin tưởng của quý khách là niềm vinh dự của Ngọc Minh!</div>

1 0

Cho thue xe 35 cho di cua khau Mong Cai - Quang Ninh
by anddyn810＠gmail.com 19 Feb '19

19 Feb '19

<div data-redactor="1">Vào những dịp nghỉ lễ hay đơn giản là dịp cuối tuần, nếu quý khách chưa tìm được một địa điểm nào thú vị để du lịch cùng gia đình và bạn bè thì tại sao không thử một lần khám phá những cửa khẩu quốc tế - nơi cửa ngõ quốc gia để trải nghiệm những cảm giác thú vị? Ngọc Minh đưa ra dịch vụ <b><i><a href="http://thuexeotongocminh.com/dia-chi-cho-thue-xe-35-cho-gia-re-uy-tin-tai-h…">Cho thuê xe 35 chỗ</a></i></b> đi cửa khẩu Móng Cái – Quảng Ninh nhằm phục vụ nhu cầu đó của khách hàng.</div> <br> <div data-redactor="1"><b><span style="color: #ff0000;" data-redactor="1">Liên hệ cho thuê xe 35 chỗ tham quan Cửa khẩu Móng Cái: 0983.22.9996 – 0967.27.9998</span></b></div> <br> <div data-redactor="1"><b>Ngọc Minh cho thuê xe 35 chỗ tham quan Cửa khẩu Móng Cái</b></div> <br> <div data-redactor="1"><i>Thông tin về lịch trình cho thuê xe 7 chỗ tham quan Cửa khẩu Móng Cái</i></div> <br> <div data-redactor="1">- Lịch trình: Đón tại Hà Nội – Cửa khẩu Móng Cái – Trả về Hà Nội</div> <br> <div data-redactor="1">- Số km dự kiến: 314km/ chiều</div> <br> <div data-redactor="1">- Cung đường đi: Qua QL5B – QL18</div> <br> <div data-redactor="1">- Thời gian di chuyển: 5 giờ 22 phút</div> <br> <div data-redactor="1"><i>Dòng xe cho thuê xe 35 chỗ tham quan Cửa khẩu Móng Cái</i></div> <br> <div data-redactor="1">Với nhiều năm hoạt động trong lĩnh vực du lịch vận tải, Ngọc Minh luôn gây được lòng tin với khách hàng bởi sự đảm bảo chất lượng xe và độ an toàn trong các chuyến đi. Quý khách hoàn toàn có thể lựa chọn dòng xe 35 chỗ tại công ty chúng tôi cho chuyến đi gia đình hay nhóm bạn của mình. Bởi dòng xe 35 chỗ tại Ngọc Minh đều là xe đời mới, nội thất đầy đủ, sang trọng, mang lại cảm giác thoải mái nhất cho khách hàng. Hiện tại chúng tôi đang có các loại xe 35 chỗ như sau:</div> <br> <div data-redactor="1">- Dòng xe 35 chỗ Isuzu Samco</div> <br> <div data-redactor="1">- Dòng xe 35 chỗ Thaco Town</div> <br> <div data-redactor="1"><img src="https://lh3.googleusercontent.com/-Qbi8Q4-NCME/XGuvaoeP5mI/AAAAAAAAFRs/YAhb…" alt="[IMG]" /></div> <div data-redactor="1">Ngọc Minh cho thuê xe 35 chỗ đi Móng Cái</div> <br> <div data-redactor="1"><i>Giá các dòng xe 35 chỗ cho thuê tham quan Cửa khẩu Móng Cái</i></div> <br> <div data-redactor="1">Đa số khách hàng đều cho rằng, xe đời mới thì mức giá cho thuê chắc cũng sẽ rất đắt. Nhưng nếu quý khách sử dụng dịch vụ thuê xe tại Ngọc Minh, chúng tôi cam kết sẽ mang tới mức giá hợp lý so với thị trường chung. Hiện tại giá của dòng xe 35 chỗ của công ty là 11.000đ/km.</div> <br> <div data-redactor="1">Tham khảo thêm: <b><i><a href="http://thuexeotongocminh.com/thue-xe-du-lich/thue-xe-45-cho/">Cho thuê xe 45 chỗ giá rẻ tại Hà Nội</a></i></b></div> <br> <div data-redactor="1"><b>Vài nét về Cửa khẩu Móng Cái – Quảng Ninh</b></div> <br> <div data-redactor="1">Cửa khẩu Móng Cái là một trong những cửa khẩu đầu tiên của Việt Nam. Nằm tại phường Hòa Lạc, thành phố Móng Cái, tỉnh Quảng Ninh, cửa khẩu này là nơi thông thương với cửa khẩu Đông Hưng, tỉnh Quảng Tây, Trung Quốc.</div> <br> <div data-redactor="1">Đây là một trong những điểm tham quan rất được ưa chuộng đối với du khách. Ghé thăm cửa khẩu Móng Cái, du khách chỉ cần đăng ký với bên biên phòng là có thể vào khu vực cửa khẩu để chụp ảnh cùng mốc 1369 cũng như vạch biên giới nằm trên cầu Hữu Nghị Bắc Luân.</div> <br> <div data-redactor="1">Ngoài ra, nếu muốn khám phá thành phố Đông Hưng, Trung Quốc, du khách hãy chuẩn bị sẵn 2 ảnh 4x6, CMND và giấy khai sinh nếu có trẻ em dưới 15 tuổi để làm sổ thông hành. Thủ tục khá nhanh gọn chỉ mất khoảng 1 tiếng đồng hồ là bạn có thể xuất cảnh qua cửa khẩu rồi.</div> <br> <div data-redactor="1">Và nếu đã đặt chân tới Móng Cái thì đừng quên ghé qua chợ Móng Cái để có thể mua được những món đồ vừa ý về làm quà cho gia đình và bạn bè nhé!</div> <br> <div data-redactor="1">Còn chần chừ gì nữa mà không liên hệ ngay với <a href="https://www.facebook.com/chothuexedulichngocminh/"><b><i>Cho thuê xe 35 chỗ Hà Nội</i></b> </a>để có một chuyến đi hoàn hảo, đảm bảo an toàn và chất lượng tuyệt đối? Hãy gọi vào hotline <b><i><span style="color: #ff0000;" data-redactor="1">0983.22.9996 – 0967.27.9998</span></i></b> để được tư vấn cụ thể về lịch trình cũng như cách thức thuê xe.</div> <br> <div data-redactor="1">Sự tin tưởng của quý khách là niềm vinh dự của Ngọc Minh!</div>

1 0