Author Topic: Another question about _MEM...  (Read 146 times)

DSMan195276

  • Hero Member
  • *****
  • Posts: 1973
  • Yes
    • Email
Another question about _MEM...
« on: April 12, 2013, 09:03:27 PM »
Ok, so some example code:

Code: [Select]
DIM m1 AS _MEM, m2 AS _MEM

s = 5
m1 = _MEM(s)
m2 = m1

_MEMFREE m1

'print _MEMGET(m1, m1.OFFSET, SINGLE)
PRINT _MEMGET(m2, m2.OFFSET, SINGLE)

END

The above code prints '5'. If you uncomment the other print line, then it will crash. It gets weirder though, this doesn't work:

Code: [Select]
DIM m1 AS _MEM, m2 AS _MEM

m1 = _MEMNEW(4)
m2 = m1

_MEMPUT m1, m1.OFFSET, 5 AS LONG

_MEMFREE m1

'print _MEMGET(m1, m1.OFFSET, long)
PRINT _MEMGET(m2, m2.OFFSET, long)

END

Which, you'd expect, as the reason for it crashing is much more obvious. In the second case I'm allocating some new memory and when I free m1 the memory is gone. So when m2 tries to reference it nothing is there to grab anything (If someone could tell me what happens with the second one on Windows, that would be nice. I suspect you won't get a QB64 error but the program will just crash).

But, now again we have the question of what does _MEMFREE do and what does making a copy of _MEM's just using '=' do. From what Galleon has explained, the internal flags of _MEM should have cause errors in both cases. Making a copy of one _MEM to another I presumed would keep the internal stuff the same. And that is were the issues come in.

In the first example, QB64 won't complain if you '_MEMFREE m2' (after the print that reads it's contents of course), and that tells me that 'm2' is a complete, new, and valid _MEM which is separate from m1 internally but still references the same location, so just setting one _MEM equal to another does more then just set the values equal. And this is a *huge* problem. Because that means that there is now some internal stuff that needs to be deallocated to prevent a memory leak from _MEMs internals, but at the same time we've made a situation were multiple _MEM's point to the same location, and when _MEMFREE is used they all try to deallocate that same location. This problem is masked in situations where the _MEM is not pointing to some newly allocated memory (Because in that case _MEMFREE doesn't actually try to deallocate memory), however in cases were memory was allocated then only one of the _MEM's you have can be deallocated without creating an error. And so this means you're forced to have a memory leak.

Even in cases where you didn't use _MEMNEW, and thus can free every _MEM, that means you have to free every single copy of a _MEM you create, leading to countless easy to make memory leaks (Every time you make a copy of a _MEM using '=', you have to free it).

Some ideas to fix it: _MEMFREE could split into two commands, one to free the _MEM and one to free memory allocated from _MEMNEW. That would at least avoid the situation of unavoidable memory leaks, but of course would put memory leaks into pretty much all _MEM code written before now. It woudl however make everything about _MEM much clearer and probably easier to use correctly (At the cost of requiring more free statements). Another option is to (Like talked about before in the last topic) make it so that only memory allocated by _MEMNEW has to be freed in the first place. That fixes it because then copies of _MEM's won't have to be freed anyway. One one of the _MEM's pointing to a _MEMNEW will have to be freed to prevent memory leaks. Or the other option would be to make sure all internal stuff is kept constant between _MEM copies (So freeing one frees the internal state of the other). While it'd be hard to also invalidate the data the other copies of the _MEM contain after one copy is freed (So they'll contain invalid _OFFSET values after the free, like m2 in the second example), at least in this case the copies wouldn't need to be freed to prevent a memory leak (Just don't use any copies after one of the copies is freed). Of course, any/all of these will create minor to major compatibility issues, but I don't think leaving it like this is really an option because it just creates memory leaks.

Any other ideas or thoughts?

Matt
"Cast your cares on the Lord and he will sustain you; he will never let the righteous be shaken" -- Psalm 55:22
QB64 Linux Installer

SkyCharger001

  • Hero Member
  • *****
  • Posts: 1591
Re: Another question about _MEM...
« Reply #1 on: April 13, 2013, 01:03:51 AM »
_memfree causes m1 to become a null-pointer.
it's like you're trying to access file #1 AFTER you closed it

DSMan195276

  • Hero Member
  • *****
  • Posts: 1973
  • Yes
    • Email
Re: Another question about _MEM...
« Reply #2 on: April 13, 2013, 01:27:19 AM »
_memfree causes m1 to become a null-pointer.
it's like you're trying to access file #1 AFTER you closed it

I understand that, however keep in mind _MEMFREE does more then just make m1 into a null pointer. It's easy enough to set an _OFFSET to 0 so if that's all _MEMFREE did then freeing _MEM besides _MEMNEW wouldn't be necessary. _MEMFREE at the very least also releases some internal locks on the _MEM that QB64 uses. And then you have the fact that if you set 'm2 = m1' you can still use m2 after m1 is freed, indicating that whatever internal locks are released, every copy of a _MEM get's a new set of locks but the same reference (And these locks are what can create a memory leak, because if you don't free the locks then you start eating up memory). And this is more obvious by the fact that the first code is valid but the second one is not. The only difference is that in the second, the memory that is referenced is actually freed were as the first one, the reference is still valid even if _MEMFREE was used on it.

What I'm pointing out here is similar to if you had to use 'close' on every numeral variable you stored file number in to prevent a memory leak, but using 'close' on more then one results in an error.

Matt
"Cast your cares on the Lord and he will sustain you; he will never let the righteous be shaken" -- Psalm 55:22
QB64 Linux Installer

SMcNeill

  • Hero Member
  • *****
  • Posts: 2414
    • Email
Re: Another question about _MEM...
« Reply #3 on: April 13, 2013, 02:13:52 AM »
Ok, so some example code:

Code: [Select]
DIM m1 AS _MEM, m2 AS _MEM

s = 5
m1 = _MEM(s)
m2 = m1

_MEMFREE m1

'print _MEMGET(m1, m1.OFFSET, SINGLE)
PRINT _MEMGET(m2, m2.OFFSET, SINGLE)

END

The above code prints '5'. If you uncomment the other print line, then it will crash. It gets weirder though, this doesn't work:


This works as I'd expect.   M1 is a pointer to the variable s.   you make m2 a pointer to the variable s.  then you discard the pointer m1.  At this point you now have s and m2 as a pointer to s.   which is why it prints 5.  Try and print m1, and it's freed and gone, and errors out.

Quote
Code: [Select]
DIM m1 AS _MEM, m2 AS _MEM

m1 = _MEMNEW(4)
m2 = m1

_MEMPUT m1, m1.OFFSET, 5 AS LONG

_MEMFREE m1

'print _MEMGET(m1, m1.OFFSET, long)
PRINT _MEMGET(m2, m2.OFFSET, long)

END

Which, you'd expect, as the reason for it crashing is much more obvious. In the second case I'm allocating some new memory and when I free m1 the memory is gone. So when m2 tries to reference it nothing is there to grab anything (If someone could tell me what happens with the second one on Windows, that would be nice. I suspect you won't get a QB64 error but the program will just crash).

And this too behaves as you'd expect.  M1 created a new memory area, m2 pointer to it, and then m1 got rid of it.  M2 now points to nothing....

I don't see these as flaws in the _MEM command, but instead flaws in the programmers use of the commands.  Honestly, they seem to be acting as I'd expect they should.
http://bit.ly/TextImage -- Library of QB64 code to manipulate text and images, as a BM library.
http://bit.ly/Color32 -- A set of color CONST for use in 32 bit mode, as a BI library.

http://bit.ly/DataToDrive - A set of routines to quickly and easily get data to and from the disk.  BI and BM files

Galleon

  • Administrator
  • Hero Member
  • *****
  • Posts: 4663
  • QB Forever
    • Email
Re: Another question about _MEM...
« Reply #4 on: April 13, 2013, 05:12:54 AM »
All of the behaviour is by design. By "crash" you are referring to the error message "Memory has been freed" right? Because that is what I get and that is what it should return. Think of the _MEM locking system as an agreement between the _MEM structure and an internal flag stating that object's memory exists. If either doesn't check out something is wrong, so it gives this error.

For efficiency, QB64 will only store one flag per scope for static (at least static in memory) items within that scope. This means all your stock standard variables inside the main section of your program are using a single internal lock. No matter how many _MEM structures are referring to how many offsets of variables you are using the same singular internal lock. You've probably noticed by now that every SUB/FUNCTION creates a scope lock on entry and clears it on return, even if there's nothing which references using _MEM within that scope. I do intend to omit the creation of redundant locks in the future, just hasn't been a high priority.

In the case of your second example, the scope of that memory isn't bound to the 'main' scope of the code, so QB64 tracks its lock individually.

What you are don't in the first example is a minor abuse of the _MEM system. It certainly doesn't make sense to refer to a reference which has been freed. Why would you? The QB64 _MEM system protects you from segfaults/GPFs, not hack programming.
Something old... Something new... Something borrowed... Something blue...

mcalkins

  • Hero Member
  • *****
  • Posts: 1265
    • qbasicmichael.com
    • Email
Re: Another question about _MEM...
« Reply #5 on: April 13, 2013, 05:38:55 AM »
So, basically, _MEMFREEing any one copy of the _MEM is sufficient, and none of the other copies need to be freed.

I suppose, that if someone has a really complex program, where _MEMs and copies of _MEMs get passed around, how do you know when to free it? If part of the program frees it too early, other parts will be accessing an invalid _MEM. If nobody frees it, you might be leaking something. In such a case, you could add a reference count:

Code: [Select]
TYPE RefCount
 m AS _MEM
 count AS _UNSIGNED LONG
END TYPE

Each part of the program that could potentially need to free the memory would first decrement the count. Then, if and only if the count is zero, it would call _MEMFREE. Each time a new copy is made, and passed to such a part of the program, the count would be incremented. (You can check for overflow if appropriate.) The count would start at 1 when you initialize the _MEM.

P.S. There would be a potential problem if you tried to make copies of the RefCount. If you had 2 separate copies of it, then you would be incrementing/decrementing 2 separate variables. You would need to pass the RefCount byref rather than making copies of it. So, you wouldn't actually be making copies of the _MEM either. But it would still be useful for the situation that I described. Alternatively, you could maintain, a separate count variable in the scope where the _MEM was created, and the RefCount would contain its _OFFSET. Then you could copy the RefCount. This could have an efficiency advantage, because there would be 1 fewer indirection from accessing the _MEM, though you would still have an indirection from accessing the counter.

P.S. On third thought, rather than putting them in the same structure, you could just pass them both together. Pass the _MEM and pass the count by reference. They don't have to be in the same structure.

Regards,
Michael
« Last Edit: April 13, 2013, 06:08:05 AM by mcalkins »
The QBASIC Forum Community: http://www.network54.com/index/10167 Includes off-topic subforums.
QB64 Off-topic subforum: http://qb64offtopic.freeforums.org/

Galleon

  • Administrator
  • Hero Member
  • *****
  • Posts: 4663
  • QB Forever
    • Email
Re: Another question about _MEM...
« Reply #6 on: April 13, 2013, 06:55:50 AM »
Quote
So, basically, _MEMFREEing any one copy of the _MEM is sufficient, and none of the other copies need to be freed.
Correct. It depends how you define 'copy' though.

This is correct:
Code: [Select]
s&=123
dim m1 as _mem, m2 as _mem
m1=_mem(s&)
m2=_mem(s&)
_memfree m1
_memfree m2

This is also correct:
Code: [Select]
s&=123
dim m1 as _mem, m2 as _mem
m1=_mem(s&)
m2=m1
_memfree m1 'or _memfree m2, but not both!

This is wrong (but won't actually leak in the current implementation):
Code: [Select]
s&=123
dim m1 as _mem, m2 as _mem
m1=_mem(s&)
m2=_mem(s&)
_memfree m1

Think of a _MEM structure as a handle for memory. If you OPEN the file twice, you need to close it twice, but if you merely move the value of one handle into another variable it is still the same handle.
« Last Edit: April 13, 2013, 07:11:50 AM by Galleon »
Something old... Something new... Something borrowed... Something blue...

DSMan195276

  • Hero Member
  • *****
  • Posts: 1973
  • Yes
    • Email
Re: Another question about _MEM...
« Reply #7 on: April 13, 2013, 10:42:28 AM »
Perhaps it was partly because I was in Linux (Linux doesn't seem to report memory errors, the programs currently just crash). My point here is that in the second example you gave, freeing m2 is completely valid, it gives me no error at all *unless* the memory was created by _MEMNEW (Because the 'free' call fails):

Code: [Select]
DIM m1 AS _MEM, m2 AS _MEM

s = 5
m1 = _MEM(s)
m2 = m1

_MEMFREE m1

'print _MEMGET(m1, m1.OFFSET, SINGLE)
PRINT _MEMGET(m2, m2.OFFSET, SINGLE)

_MEMFREE m2

print "M2 freed!"

END

What I'm saying is that if freeing m2 is still valid, why? If it was indeed done with then I would have assumed it would have been invalided internally when m1 was freed, unless it has it's own internal state (Which then by that assumption, it would also have to have it's internal state freed.).

It *should* create an error shouldn't it? The only way I see that it could not create an error is if m2 then is it's own _MEM separate from m1 (And thus, both m1 and m2 need to be freed to conserve memory, barring anything that QB64 may do internally). If they were not separate, then the program should have crashed. And again, this seems to be more obvious by the fact that freeing m2 is valid in the code.

Here, this program creates a simple linked list type of structure with _MEM:

Code: [Select]
TYPE stack_link
  dat AS INTEGER
  n_ele as _MEM 'Next element
END TYPE

DIM start as _MEM

DO: _limit 10000
  for x = 1 to 255
  push_stack start, x
  next x
 
  for x = 1 to 255
    print pop_stack%(start),
  next x
loop until inkey$ > ""

END

SUB push_stack (stack as _MEM, n as INTEGER)
DIM s as stack_link

s.dat = n

s.n_ele = stack
stack = _MEMNEW(len(s))

_MEMPUT stack, stack.OFFSET, s
end sub

FUNCTION pop_stack% (stack as _MEM)
DIM s as stack_link

_MEMGET stack, stack.OFFSET, s
_MEMFREE stack

stack = s.n_ele

pop_stack% = s.dat
END FUNCTION

So the structure works by just keeping track of the top link of the list. etc. The point here is that I use _MEMNEW() to allocate a new spot on the list, and then I put that into stack. Right before that, I set s.n_ele equal to the stack (So I'm saving the _MEM to the top link). But when I do that, do I just end-up creating *another* _MEM which has it's own internal locks but has the same _OFFSET and points to the same value? And if so, then when I set stack equal to _MEMNEW() it should technically have a memory leak because I never freed the internal locks on stack _MEM before overwriting it's information? It doesn't have a memory leak, but is that because QB64 does some clean-up at the end for me or is it because there was no new internal states for s.n_ele?

I don't want to introduce any memory leaks into my programs, and that requires careful attention to what's going on with them, however if every time I set one _MEM equal to anther it adds more internal stuff then I don't see how memory leaks from that could be avoided. I realize QB64 does do it's own clean-up at various point, but relying on that when it's not completely understood by anybody except Galleon seems like a bad idea and is just asking to introduce memory leaks unintentionally where you assume it's getting taken care of but it's not. So I'm just trying to understand what's going on her and why this works the way it does.

Matt
"Cast your cares on the Lord and he will sustain you; he will never let the righteous be shaken" -- Psalm 55:22
QB64 Linux Installer

Galleon

  • Administrator
  • Hero Member
  • *****
  • Posts: 4663
  • QB Forever
    • Email
Re: Another question about _MEM...
« Reply #8 on: April 13, 2013, 03:54:03 PM »
After a brief look at your program, I can see there is a 1 to 1 relationship between the number of times _MEMNEW is called and the number of times _MEMFREE is called. Therefore, your program is correct and will not leak.
In all uses of _MEM, there should be a 1 to 1 relationship between _MEMNEW(...)/_MEM(...)/_MEMIMAGE/etc and _MEMFREE.
Something old... Something new... Something borrowed... Something blue...

DSMan195276

  • Hero Member
  • *****
  • Posts: 1973
  • Yes
    • Email
Re: Another question about _MEM...
« Reply #9 on: April 13, 2013, 04:18:42 PM »
After a brief look at your program, I can see there is a 1 to 1 relationship between the number of times _MEMNEW is called and the number of times _MEMFREE is called. Therefore, your program is correct and will not leak.
In all uses of _MEM, there should be a 1 to 1 relationship between _MEMNEW(...)/_MEM(...)/_MEMIMAGE/etc and _MEMFREE.

Fair enough. However that doesn't explain this:

Code: [Select]
DIM m1 as _MEM, m2 as _MEM
s = 54

m1 = _MEM(s)
m2 = m1

_MEMFREE m1
_MEMFREE m2

print "test"

end

I realize that in the sense of what you said it's not the proper way to do it, but why does it work? If this was a pointer in C, then this would completely blow up because you tried to free the same data two twice. I realize _MEM's are not pointers, but similarity, I would have assumed that if I just set one _MEM variable equal to another _MEM, I would only be able to free one without an error (But conversely, only one would be required to be freed to prevent the memory leak so the fact that only one is freeable is a non-issue). Because I can fee both without error, doesn't that mean that at the very least they have separate internals?

Matt
"Cast your cares on the Lord and he will sustain you; he will never let the righteous be shaken" -- Psalm 55:22
QB64 Linux Installer

Galleon

  • Administrator
  • Hero Member
  • *****
  • Posts: 4663
  • QB Forever
    • Email
Re: Another question about _MEM...
« Reply #10 on: April 13, 2013, 08:39:56 PM »
Simple explanation: It's doing something you shouldn't but QB64 has no system in place (for efficiency reasons) to let you know what you are trying to do is wrong. Inside the _MEM type structure it sets variables differently after you have freed the reference. In the case of the code above, the actual memory still exists, and reference m2 still points to it, therefore QB64 has no way of knowing it needs to give an error.
Something old... Something new... Something borrowed... Something blue...

DSMan195276

  • Hero Member
  • *****
  • Posts: 1973
  • Yes
    • Email
Re: Another question about _MEM...
« Reply #11 on: April 13, 2013, 08:53:01 PM »
So then to be clear, even though freeing m2 works, _MEMFREE basically just ignores the _MEM and doesn't throw an error? And more importantly, m2 isn't taking up any memory internally that _MEMFREE didn't already previously get rid of when m1 was freed?

Matt
"Cast your cares on the Lord and he will sustain you; he will never let the righteous be shaken" -- Psalm 55:22
QB64 Linux Installer