Foreword: This article by IBM's Sachin Agrawal discusses the Linux operating system's shared memory architecture, and how object-oriented C++ programmers can best exploit it. According to Agrawal, shared memory is the most efficient inter-process communication channel available in modern operating systems, and Linux's shared memory architecture is uniquely useful.

In terms of time and space, shared memory is probably the most efficient inter-process communication channel provided by all modern operating systems. Shared memory is simultaneously mapped to the address space of more than one process: a process simply attaches to the shared memory and starts communicating with other processes by using it as it would use ordinary memory.

However, in the object-oriented programming world, processes prefer to share objects rather than raw information. With objects, there is no need to serialize, transport, and de-serialize the information contained within the object. Shared objects also reside in shared memory, and although such objects “belong” to the process that created them, all processes on the system can access them. Hence, all of the information within a shared object should be strictly process-neutral.

This is a direct contradiction to the C++ object model currently adopted by all popular compilers: C++ objects invariably contain pointers to various Vee-Tables and sub-objects that are process-specific. For such objects to be sharable, you need to make sure that the target of these pointers resides at the same address in all the processes.

With the help of a small sample program, this article illustrates cases where the C++ model succeeds, where it fails to work with the shared memory model, and where possible workarounds exist. The discussion and the sample program are limited to non-static data members and virtual functions. Other situations are not as relevant to the C++ object model as these are: static and non-static non-virtual member functions do not have any issues in shared environment. Per-process static members do not reside in shared memory (and thus have no issues), while shared static members have issues similar to those discussed here.

Environmental assumptions

This text is highly specific to the Red Hat Linux 7.1 distribution for 32-bit x86 Intel architectures, as GNU C++ compiler version 2.95 and associated tools were used to build and test the sample program. However, you can apply the overall concepts equally well to any machine architecture, operating system, and compiler combinations.

Sample program

The sample program consists of two clients, shm_client1 and shm_client2, and uses the shared objects services provided by the shared library shm_server. Object definitions reside in common.h:

Listing 1. Definitions in common.h

   #ifndef __COMMON_H__
   #define __COMMON_H__

   class A {
   public:
       int m_nA;
       virtual void WhoAmI();

       static void * m_sArena;
       void * operator new (unsigned int);
   };


   class B : public A {
   public:
       int m_nB;
       virtual void WhoAmI();
   };


   class C : virtual public A {
   public:
       int m_nC;
       virtual void WhoAmI();
   };

   void GetObjects(A ** pA, B ** pB, C ** pC);

   #endif //__COMMON_H__

Listing 1 defines three classes (A, B and C) with a common virtual function WhoAmI(). The base class, A, has a member named m_nA. The static member m_sArena and overloaded operator new() are there to enable the construction of objects in shared memory. Class B is simply derived from A, and class C is virtually derived from A. Both B::m_nB and C::m_nC are provided to ensure that the sizes of A, B, and C are distinct. This simplifies the implementation of A::operator new(). The interface GetObjects() returns the shared objects pointers.

The shared library implementation is in shm_server.cpp:

Listing 2. Library – shm_server.cpp

   #include sys/types.h>
   #include sys/ipc.h>

   #include sys/shm.h>
   #include errno.h>
   #include stdio.h>
   #include iostream>

   #include "common.h"

   void * A::m_sArena = NULL;

   void * A::operator new (unsigned int size)
   {
       switch (size)
       {
       case sizeof(A):
           return m_sArena;

       case sizeof(B):
           return (void *)((int)m_sArena + 1024);

       case sizeof(C):
           return (void *)((int)m_sArena + 2048);

       default:
           cerr  __FILE__  ":"  __LINE__  " Critical error"  
endl;
       }
   }

   void A::WhoAmI() {
       cout  "Object type: A"  endl;
   }


   void B::WhoAmI() {
       cout  "Object type: B"  endl;
   }


   void C::WhoAmI() {
       cout  "Object type: C"  endl;
   }


   void GetObjects(A ** pA, B ** pB, C ** pC) {
       *pA = (A *)A::m_sArena;
       *pB = (B *)((int)A::m_sArena + 1024);
       *pC = (C *)((int)A::m_sArena + 2048);
   }


   class Initializer {
   public:
       int m_shmid;
       Initializer();

       static Initializer m_sInitializer;
   };

   Initializer Initializer::m_sInitializer;

   Initializer::Initializer()
   {
       key_t key = 1234;
       bool bCreated = false;

       m_shmid = shmget(key, 3*1024, 0666);

       if (-1 == m_shmid) {

           if (ENOENT != errno) {
               cerr  __FILE__  ":"  __LINE__  " Critical error"  

endl;
               return;
           }

           m_shmid = shmget(key, 3*1024, IPC_CREAT | 0666);

           if (-1 == m_shmid) {
               cerr  __FILE__  ":"  __LINE__  " Critical error" 
endl;
               return;
           }

           cout  "Created the shared memory"  endl;
           bCreated = true;
       }

       A::m_sArena = shmat(m_shmid, NULL, 0);

       if (-1 == (int)A::m_sArena) {
           cerr  __FILE__  ":"  __LINE__  " Critical error" 

endl;
           return;
       }

       if (bCreated) {
           // Construct objects on the shared memory
           A * pA;
           pA = new A;
           pA->m_nA = 1;
           pA = new B;
           pA->m_nA = 2;
           pA = new C;
           pA->m_nA = 3;
       }

       return;
   }

Let's look at Listing 2 in more detail:

Lines 9-25: operator new ()

The same overloaded operator allows you to construct objects of class A, B, and C in shared memory. Object A starts right at the beginning of shared memory. Object B starts at offset 1024, and C at offset 2048.

Lines 26-34: Virtual functions

The virtual functions simply write a line of text to standard output.

Lines 35-39: GetObjects

GetObjects() returns the pointers to shared objects.

Lines 40-46: Initializer

This class stores the shared memory identifier. Its constructor creates the shared memory and objects in it. If the shared memory already exists, it simply attaches to it. The static member m_sInitializer ensures that the constrictor is invoked prior to the main() function of the client module that is using the shared library.

Lines 48-82: Initializer::Initializer()

Shared memory is created if it does not exist, and shared objects are created within it. The object construction is skipped if the shared memory already exists. Initializer::m_shmid records the identifier and A::m_sArena records the shared memory address.

The shared memory is not destroyed even after all processes detach from it. This allows you to explicitly destroy it using the ipcrm command or to make some quick observations with the ipcs command.

The client process implementation is in shm_client.cpp:

Listing 3. Client – shm_client.cpp

   #include "common.h"

   #include iostream>

   #include stdlib.h>

   int main (int argc, char * argv[])
   {
       int jumpTo = 0;

       if (1  argc) {
           jumpTo = strtol(argv[1], NULL, 10);
       }

       if ((1 > jumpTo) || (6  jumpTo)) {
           jumpTo = 1;
       }

       A * pA;
       B * pB;
       C * pC;

       GetObjects(&pA, &pB, &pC);

       cout  (int)pA  "t";
       cout  (int)pB  "t";
       cout  (int)pC  " ";

       switch (jumpTo) {
       case 1:
           cout  pA->m_nA  endl;

       case 2:
           pA->WhoAmI();

       case 3:
           cout  pB->m_nA  endl;

       case 4:
           pB->WhoAmI();

       case 5:
           cout  pC->m_nA  endl;

       case 6:
           pC->WhoAmI();
       }

       return 0;
   }

   #include pthread.h>

   void DoNothingCode() {
       pthread_create(NULL, NULL, NULL, NULL);
   }

Lines 6-35

The client process gets pointers to three shared objects, makes three references to their data members, and — depending on the command-line input — invokes three virtual functions.

Lines 36-39

The uninvolved pthread_create() function is used to force linking with another shared library. Any function from any shared library will serve this purpose.

The shared library and two instances of client executable are built as follows:

gcc shared g shm_server.cpp o libshm_server.so lstdc++
gcc -g shm_client.cpp -o shm_client1 -lpthread -lshm_server -L .
gcc -g shm_client.cpp -o shm_client2 -lshm_server -L . lpthread

Note that the link sequence of shm_server and pthread for shm_client1 and shm_client2 is swapped to ensure that the base address of the shm_server shared library in both executables is different. This can further be verified using the ldd command. Sample output is usually something like the following:

Listing 4. Library map for shm_client1

ldd shm_client1

libpthread.so.0 => (0x4002d000)
libshm_server.so => (0x40042000)
libc.so.6 => (0x4005b000)
ld-linux.so.2 => (0x40000000)

Listing 5. Library map for shm_client2

ldd shm_client2

libshm_server.so => (0x40018000)
libpthread.so.0 => (0x40046000)
libc.so.6 => (0x4005b000)
ld-linux.so.2 => (0x40000000)

The main aim here is to build two client binaries that have distinct base addresses for the server library. In the context of this sample program, using the uninvoked pthread_create() function and the different link sequence for shared libraries works to accomplish this goal. However, there is no concrete rule or uniform procedure available that works across all linkers; different methods will need to be employed on a case-by-case basis.

Case 1: shm_client1 vs. shm_client1

In the following output, the shm_client1 is invoked first from the shell. Because no shared objects are present, it creates them, references their data members, invokes their virtual functions, and quits — leaving the objects behind in memory. The second time, the process simply references the data members and virtual functions.

Listing 6. Output log for shm_client1 vs. shm_client1

   $ ./shm_client1

   Created the shared memory
   1073844224 1073845248 1073846272
   1
   Object type: A
   2
   Object type: B
   3
   Object type: C


   $ ipcs

   ------ Shared Memory Segments --------

   key shmid owner perms bytes nattch status
   0x000004d2 2260997 sachin 666 3072 0


     $ ./shm_client1
     1073840128 1073841152 1073842176
     1
     Object type: A
     2
     Object type: B
->   0
->   Segmentation fault (core dumped)

When the second process tries to refer to data member A::m_nA via the pointer of type C * (you will remember that C is virtually derived from A), the base sub-object pointer from within the shared object is read. The shared object was constructed in the context of a now non-existing process. Hence, garbage is read for both A::m_nA and C::WhoAmI().

This is because the Vee-Table and virtual functions lie within the shm_server shared library, which happens to be reloaded at the same virtual address. Hence, no problem is observed while de-referring pointers of type A * and B *.

Hence, the C++ object model adopted by GNU fails with virtual inheritance.

Case2: shm_client1 vs. shm_client2

In the next sample output, first the shm_client1 and then the shm_client2 are executed from the command line:

Listing 7. Output log for shm_client1 vs. shm_client2

   $ ./shm_client1

   Created the shared memory
   1073844224 1073845248 1073846272
   1
   Object type: A
   2
   Object type: B
   3
   Object type: C


   $ ipcs

   ------ Shared Memory Segments --------
   key shmid owner perms bytes nattch status
   0x000004d2 2359301 sachin 666 3072 0


   $ ./shm_client2
   1073942528 1073943552 1073944576
   1
-> Segmentation fault (core dumped)

   $ ./shm_client2 3
   1073942528 1073943552 1073944576
   2
-> Segmentation fault (core dumped)

   $ ./shm_client2 5
   1073942528 1073943552 1073944576
-> 1048594
-> Segmentation fault (core dumped)

However, the Vee-Table lies within the shm_server shared library: it is loaded to different virtual addresses within shm_client1 and shm_client2. Hence, garbage is read for both A::WhoAmI() and B::WhoAmI().

Making shared memory work

You should consider two major issues when instantiating C++ objects within shared memory. First, the Vee-Table pointer is used to access virtual functions, and data members are accessed directly using compile time offsets. Hence, for all such shared objects, Vee-Table and virtual functions should have the same virtual addresses in all processes. There is no hard-and-fast rule for doing this, but adopting a proper link sequence for dependent shared libraries should work most of the time.

In addition, keep in mind that virtually inherited objects have base pointers to address base objects. Base pointers refer to data sections of the process, and are always process specific. It is difficult to ensure the same numerical values for these across all client processes. Hence, construction of virtually inherited objects in shared memory should be avoided for the assumed C++ object model. But keep in mind, too, that different compilers adopt different models. For instance, Microsoft Compiler uses process-neutral offsets to address base objects for virtually inherited classes: thus, this issue does not arise. The important thing is to ensure the same addresses for shared libraries in all client-processes.

Resources

• An excellent resource for understanding C++ internals is Inside the C++ Object Model by Stanley B. Lippman (Addison-Wesley, 1996).
• RFC 1014 – XDR: External Data Representation Standard is a standard for the description and encoding of data between different computer architectures. It defines language to describe strings, variable-length arrays, and similar structures.
• Don't forget also to check the system man pages for individual functions like ld, ldd, ipcs, ipcrm and so on.
• Though Eclipse is mainly a Java development environment, its architecture ensures support for other programming languages. Learn more in the article C/C++ development with the Eclipse Platform (developerWorks, April 2003)
• IBM's VisualAge C++ is an advanced C/C++ compiler, with versions available for AIX and selected Linux distributions.
• The tutorial Java programming for C/C++ developers (developerWorks, May 2002) gives an introduction to the Java language from the viewpoint of someone already familiar with C++.
• Find more resources for Linux developers in the developerWorks Linux zone.
• Purchase Linux books at discounted prices in the Linux section of the Developer Bookstore.
• Develop and test your Linux applications using the latest IBM tools and middleware with a developerWorks Subscription: you get IBM software from WebSphere, DB2, Lotus, Rational, and Tivoli, and a license to use the software for 12 months, all for less money than you might think.
• Download no-charge trial versions of selected developerWorks Subscription products that run on Linux, including WebSphere Studio Site Developer, WebSphere SDK for Web services, WebSphere Application Server, DB2 Universal Database Personal Developers Edition, Tivoli Access Manager, and Lotus Domino Server, from the Speed-start your Linux app section of developerWorks. For an even speedier start, help yourself to a product-by-product collection of how-to articles and tech support.

About the author

Sachin Agrawal has been working extensively in C++ for five years, including three years of research into the C++ object models of various compilers. He currently works for IBM Global Services India. You can contact him via email.

First published by IBM developerWorks. Reproduced by LinuxDevices.com with permission.

 
This article was originally published on LinuxDevices.com and has been donated to the open source community by QuinStreet Inc. Please visit LinuxToday.com for up-to-date news and articles about Linux and open source.