Archive for December, 2009

Configuring ocfs2 using multipath for clustering oracle databases

Wednesday, December 16th, 2009

Here is one method for using multipath with ocfs2 on linux to create database volumes on a cluster. The reason for using multipath with ocfs2 is to bind the disks to a file system using the WWID numbers instead of allowing the operating system to assign device names. Device names can change, but WWID will not change.

  1. Make sure you have ocfs2 and multipath installed on all cluster nodes.
  2. Configure ocfs2 on the first node.
  3. Export the configuration to all nodes in the cluster. You can use ocfs2console to make life easier for your migration. You can check the nodes in the /etc/ocfs2/cluster.conf file to ensure all nodes are listed.
  4. NOTE: At this point you are finished with OCFS and be working on the local systems.
  5. Import your luns. Make sure they are shared between both or all nodes in the cluster.
  6. Set boot.multipath, multipathd, o2cb and ocfs2 to start at boot on all nodes.
  7. Configure the devices in the multipath.conf file. You can find the device names and channels using the lsscsi command. You can find the WWID with the following command: udevinfo -q symlink -n /dev/.
  8. After the devices are configured in multipath.conf run the following: multipath -F and multipath -v2. You should see your devices under /dev/mapper/.
  9. In order for ocfs2 to see the devices they need an ocfs2 file system. You can run the following foreach loop in Kshell to accomplish this:
          cd /dev/mapper
          foreach n (`ls db*`)
          sudo mkfs -t ocfs2 -N   -L $n $n
          end
  10. Create the mount points.
  11. Add the devices to /etc/fstab. Here is an example:
        /dev/mapper/      ocfs2  _netdev  0 0

    Note: the arguments for the file systems will vary. Oracle recommends _netdev,datavolume,noinitr for Oracle data volumes like raw, db, u and l. Regular file systems only get _netdev

  12. Run: sudo mount -a. This will mount the volumes locally and since there is an ocfs2 file system on them OCFS will pick them up.
  13. Change the permissions and ownership on the mounted volumes. Do not do this ahead of time, because the file system you created will over-mount the local directory.
  14. Copy multipath.conf to the other nodes and run multipath -F then multipath -v2. If you have any problems make sure multipathd is started.
  15. Create the mount points on each node.
  16. Add the entries to the local /etc/fstab.
  17. Run sudo mount -a.
  18. Do this for every node in the cluster.

Walla, you are finished and have your OCFS and multipath working.  If you have any comments of suggestions please post them. As always, please check out the sponsors on this site.

Change network interfaces in linux with tg3 from eth0 to eth1

Tuesday, December 15th, 2009

Thought I would share this, because there are many of us the need to use specific interface to support specific network traffic. One of the difficulties you have probably encountered is the network device drivers. However, just think about what is going on and you will derive a solution.

Now, when the system boots there is a hardware discovery, coldplug and hotplug processes. After that initiation the drivers start to load. The problem is the order the drivers load and not the driver itself. So, the result is that you might want a PCI NIC card to become eth2 and the on-board to become eth0 and eth1.

I was working with Novell’s SLES 9 SP4 on an IBM x336. After the system booted, I would perform an “ifconfig” command and find the devices not in the order I required, with the PCI network interface always being eth0. I added “ifdown – nameif – ifup” commands to the network init script to no avail. It was like the rc script was not being read. The problem turned out to be with the tg3 (Tiggon) driver.

The solution I came up with was very simple and you might laugh after reading this post.

  1. I kept my “ifdown – nameif – ifup” scipts intact within the rcnetwok script. You will need to down the interfaces, rename them and bring them back up again.
  2. Within the /etc/sysconfig/hardware/, I set the STARTMODE to manual.
  3. At the start of the /etc/init.d/network script, just after the required start section, I added: /sbin/hwup /etc/sysconfig/hardware/

What this does it tell the system not to bring the device up. When the network init script runs, it brings the device up. This means there will be not “holds” on the device to your “ifdown – nameif – ifup” section of the network start script will run. You can do this for any device you do not want to come up until the network init script runs.

Hope that helps. Please patronize my sponsors who make this possible. Leave a comment to let everyone know how you solved this problem and if this solution worked for you.

Enjoy!