Opened 8 years ago

Closed 8 years ago

#31 closed defect (fixed)

Memory Leak

Reported by: Freddie Akeroyd Owned by:
Priority: major Milestone: NeXus 4.0 Release
Component: Version:
Keywords: Cc: stephane.poirier@…

Description

  • Reported by POIRIER Stephane [stephane.poirier@…] *

Hi all,

The process (at SOLEIL) in charge of recording NeXus files opens, writes data and closes each NeXus file several times. Furthermore this process is intended to be never stopped. During intensives tests I have noticed a systematic and continuous increase of the memory consumption. After investigation (using Valgrind 3.2.4) the problem seems to be somewhere between the NeXus and HDF5 libraries.

Technical environment is :

  • NeXus API 3.0.0 (with memory leak fixed for time_buffer allocation in NX5Open function)
  • OS Linux RHEL4
  • gcc 3.4.4

The problem can be reproduced by this simple piece of code:

int main () {

const int nReOpen = 1000000000; int iReOpen; const char szFile[] = "files/leak_test_oc2.nxs";

NXhandle fileid; if (NXopen(szFile, NXACC_CREATE5, &fileid) != NX_OK) return 1; if( NXclose(&fileid) != NX_OK) return 1; for( iReOpen = 0; iReOpen < nReOpen; iReOpen++ ) {

if( 0 == iReOpen % 1000 )

printf("loop count %d\n", iReOpen);

if( NXopen(szFile, NXACC_RDWR, &fileid ) != NX_OK) return 1; if( NXclose(&fileid) != NX_OK ) return 1;

} return 0;

}

On a PC running linux RHEL4 with 1Go of RAM this program 'eat' 0.1% more memory every 20-25 secs. So in one or two days the process eat all the memory (I made the test) and then crash.

The attached files log_1000.txt and log_10000.txt are log files produced by Valgrind (with option --leak-check=yes) with 1000 and 10000 executions of the loop. One can see that the memory consumption is nearly linear between 1000 and 10000 loop counts. Surprisingly, setting nReOpen to 1000 (or 10000) and launch the program using Valgrind and wait for normal ending, the memory check tool will say that there is no leak, so I set nReopen constant to a huge number and break the execution (using ctrl+C) after roughly the expected count (using the printf) to get the leaks reports.

Also, using the following C++ code, I can see a abnormal memory consumption when running for a long time:

#include <napi.h> #include <sstream>

#define PSZ(s) (s).c_str()

const int nFiles = 1000000; const int nEntry = 10; const int nData = 10; int array_dims[2] = {5, 4}; short int i2_array[4] = {1000, 2000, 3000, 4000}; int iFile, iReOpen, iEntry, iData, iNXdata, iSimpleArraySize = 4; const char szFile[] = "leak_test.nxs";

int main () {

for( iFile = 0; iFile < nFiles; iFile++ ) {

strFile.Printf("files/leak_test_%03d.nxs", iFile); printf("file %s\n", PSZ(strFile)); NXhandle fileid; if (NXopen(PSZ(strFile), NXACC_CREATE5, &fileid) != NX_OK) return 1;

for( iEntry = 0; iEntry < nEntry; iEntry++ ) {

ostringstream oss;

oss << "entry_" << iEntry;

if (NXmakegroup (fileid, PSZ(oss.str()), "NXentry") != NX_OK) return 1; if (NXopengroup (fileid, PSZ(oss.str()), "NXentry") != NX_OK) return 1; for( iNXdata = 0; iNXdata < nData; iNXdata++ ) {

ostringstream oss; oss << "data_" << iNXdata; if (NXmakegroup (fileid, PSZ(oss.str()), "NXdata") != NX_OK) return 1; if (NXopengroup (fileid, PSZ(oss.str()), "NXdata") != NX_OK) return 1; for( iData = 0; iData < nData; iData++ ) {

std::ostringstream oss; oss << "i2_data_" << iData; if (NXmakedata (fileid, PSZ(oss.str()), NX_INT16, 1, &array_dims[1]) != NX_OK) return 1; if (NXopendata (fileid, PSZ(oss.str())) != NX_OK) return 1; if (NXputdata (fileid, i2_array) != NX_OK) return 1; if (NXclosedata (fileid) != NX_OK) return 1;

} if (NXclosegroup (fileid) != NX_OK) return 1;

} if (NXclosegroup (fileid) != NX_OK) return 1;

} if (NXclose (&fileid) != NX_OK) return 1;

Delete file unlink(szFile);

}

}

In this case the increase in memory consumption is more slowly, +0.1% of memory consumption in about 16 minutes. Comparing with the real recording process it seems that the memory consumption is related to the amount of recorded data.

Thanks for your attention.

°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°° Stéphane POIRIER Groupe Gestion des données Tél. : 01 69 35 93 38 SYNCHROTRON SOLEIL Bâtiment Central pièce A1.0.28 L'Orme des Merisiers Saint-Aubin - BP 48 91192 GIF-SUR-YVETTE CEDEX Site Web: <http://www.synchrotron-soleil.fr/> °°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°

Attachments (2)

log_1000.txt (6.5 KB) - added by Freddie Akeroyd 8 years ago.
Valgrind output
log_10000.txt (6.5 KB) - added by Freddie Akeroyd 8 years ago.
Valgrind output

Download all attachments as: .zip

Change History (10)

Changed 8 years ago by Freddie Akeroyd

Valgrind output

Changed 8 years ago by Freddie Akeroyd

Valgrind output

comment:1 Changed 8 years ago by Freddie Akeroyd

(In [861]) * Use -Wall on compiling with gcc/g++

  • Add an --enable-debug option to configure to turn off optimisation
  • Add in memory leak examples supplied by Stephane Poirier

Refs #31

comment:2 Changed 8 years ago by Freddie Akeroyd

  • Milestone set to NeXus 4.0

comment:3 Changed 8 years ago by Freddie Akeroyd

(In [865]) Fix a few minor memory leaks. The HDF4 interface now looks OK, but a leak remains in the HDF5 interface that seems to be related to properties created via H5Pcreate() in NX5open(). Refs #31.

comment:4 Changed 8 years ago by Freddie Akeroyd

(In [868]) Add script for running leak tests with useful valgrind arguments Refs #31

comment:5 Changed 8 years ago by Mark Koennecke

(In [869]) - Fixed memory issue reported by Stephan Poirier, refs #31

comment:6 Changed 8 years ago by Freddie Akeroyd

(In [871]) Improve test output. Refs #31.

comment:7 Changed 8 years ago by Freddie Akeroyd

  • Milestone changed from NeXus 4.0 Ready to NeXus 4.0 Release

comment:8 Changed 8 years ago by Mark Koennecke

  • Resolution set to fixed
  • Status changed from new to closed

I fixed this.

Mark Koennecke

Note: See TracTickets for help on using tickets.