• Print

Author Topic: Confusion with DLL output (Programmer Error)  (Read 2435 times)

SMcNeill

  • Moderator
  • Hero Member
  • *****
  • Posts: 6058
Confusion with DLL output (Programmer Error)
« on: May 04, 2014, 03:35:24 pm »
Me and Walter have been working on ways to add different functions into QB64, and one way to do it is to actually change the QB64 source and add new commands into it.   Several people don't seem so enthusiastic about this approach as they're afraid it may lead to bloat of the language with too many new, seldom-used commands being added.  To be honest, I'm of two completely different minds about the situation, as I think there's plenty of room for new things to be added; and yet I find myself not wanting to have to sort though an encyclopedia of documentation whenever I'm looking to do something.

I'm for it...   I'm against it...  I'm just as dysfunctional on the idea as anyone else here!   LOL!

So, I thought I'd play around with trying a different approach to adding in features that NOBODY should object to:  Creating DLL files for optional use with QB64. ;D

And, I've made GREAT progress towards working towards that goal, even if my C skills are still in development.


Here's the process so far, and I'll keep it as simple as possible for this example:

Make a new text document and name it whatever you want.  I'm going to make one and call it delme.cpp.   (Notice the cpp extension?  We're going to write this in C, so give it a cpp extension if you will.)

Now, our function is going to be as simple as one can be:

Code: [Select]
extern "C" __declspec(dllexport) float __stdcall d2r(float value) {
  return value;
}

My function here is called d2r, and we pass it a floating point value, which it simply passes directly back to us.

The same function in QB64 code would look like this:
Code: [Select]
FUNCTION d2r(value)
d2r = value
END FUNCTION

We pass it X, it simply returns to us that exact value...   As simple as a function can get, right?

**********************************

So we then compile this code into a dll.

To do that, first save the file as delme.cpp.    Go to the folder where you saved it at, and copy it and paste it into the internal/c/c_compiler/bin folder.   The location is a little different in SDL and GL, so you might need to look around a bit for it -- it's the folder with g++.exe located in it.    Copy it there.  (My folder was C:\QB64\internal\c\c_compiler\bin)

Now, we need to compile it.   

In Windows, open a CMD window and navigate to that folder and type this in:

Code: [Select]
g++ -O3 -s -shared -Wl,--enable-auto-image-base,--kill-at delme.cpp -o delme.dll
This should now make us a file called delme.dll

Congratulations!!   You have now created a windows DLL which you can use for whatever you need it for.   (Why you'd need THIS dll, which just returns the value you send it is beyond me -- except as a demo....)

****************************

Now, copy that dll file over into your main QB64 directory.    (C:\QB64 in my case)

Run QB64.exe like you always would to write a program...

And enter this test code:
Code: [Select]
DECLARE DYNAMIC LIBRARY "delme"
    FUNCTION d2r~& (BYVAL value~%&)
END DECLARE
PRINT d2r(45)
PRINT d2r(45!)
PRINT d2r(45#)
PRINT d2r(45##)
x = 45
PRINT d2r(x)

Now run it and watch the results...


*****************************

The process I've used is as simple as it can get, to get us to this point...    And yet the results here are quite confusing.

What we get as a set of results is:
Quote
1891898592
45
45
45
45

As long as I explicity define the value as SINGLE, DOUBLE, _FLOAT, or even a variable, this is giving us the output we'd expect.

BUT, if we send it a simple PRINT d2r(45), we get a result that is waaaaaaaaaay off from what I'd expect.

****************************
****************************

For those of you that are better at C than I am, could you look and see if I did something wrong here??   Is the issue with the output something that *I* am doing wrong?

Or is there a glitch with how QB64 handles sending direct values to a library like this?  If so, that seems like it would be a bug which should be addressed ASAP, as it could generate false results with ANY external dll which we might want to link to it for use.   

EDIT:  Nope.   (I don't guess.)   Darth got me sorted out below.   And made it look sooo easy to boot!    :-[


And, to simplify things for people who just want to test the DLL in QB64, I'm attaching the compiled version below as well.  Just download it, move it into your QB64 folder, and run the test code provided.

I can't understand for the life of me why we'd send it 45 and get back 1891898592......   And yet 45!, 45#, 45##, x all work as you'd expect.....

Bug?  Or Programmer Error??
« Last Edit: May 04, 2014, 05:30:37 pm by SMcNeill »
http://bit.ly/TextImage -- Library of QB64 code to manipulate text and images, as a BM library.

http://bit.ly/Color32 -- A set of color CONST for use in 32 bit mode, as a BI library.

DarthWho

  • Hero Member
  • *****
  • Posts: 4039
  • Timelord of the Sith
Re: Confusion with DLL output -- QB64 Bug or Programmer Error
« Reply #1 on: May 04, 2014, 05:16:05 pm »
I'm confused as to why you are assigning the output of the function as an _UNSIGNED LONG and the Value in as an _UNSIGNED OFFSET, both of which I must point out are integer types (_OFFSET by necessity must be an integer type) to a function that is declared in terms of floats.
Here is a modified version of your code that works just fine with your DLL:
Code: [Select]
DECLARE DYNAMIC LIBRARY "delme"
    FUNCTION d2r (BYVAL value AS SINGLE)
END DECLARE
PRINT d2r(45)
PRINT d2r(45!)
PRINT d2r(45#)
PRINT d2r(45##)
x = 45
PRINT d2r(x)

Tip: Keep similar types together.

I don't know why you were getting any right answers. To me that's the bigger mystery and is probably part of some safety feature Galleon implemented at some point but I have no clue.
« Last Edit: May 04, 2014, 05:21:07 pm by DarthWho »
FastMath 1.1.0 released: http://dl.dropbox.com/u/12359848/fastmath.h

BTC: 1DGmy7rBZ15Y1nFJXkoE8BkvmMu6DxSMM4
LTC: LRNzAapRvQEuuEGwuLTG1f6nuHaf7tqkn7

SMcNeill

  • Moderator
  • Hero Member
  • *****
  • Posts: 6058
Re: Confusion with DLL output -- QB64 Bug or Programmer Error
« Reply #2 on: May 04, 2014, 05:27:33 pm »
Gah!   I figured it was something simple, but I couldn't seem to put two and four together to make 3...   LOL!

I tried something quite similar, and it wouldn't work so I thought I'd pass the offsets back and forth to see if it'd work.  It *almost* did, except for the number by itself.

Here's what I tried at first, and this too tosses out issues:

Code: [Select]
DECLARE DYNAMIC LIBRARY "delme"
    FUNCTION d2r (value)
END DECLARE
PRINT d2r(45)
PRINT d2r(45!)
PRINT d2r(45#)
PRINT d2r(45##)
x = 45
PRINT d2r(x)

The glitch?  I didn't pass value BYVAL as you did...

Does anyone have a good set of instructions on WHEN byval is needed?  WHY?   And does anyone have a good writeup on the differences in the CUSTOMTYPE, STATIC, and DYNAMIC usage?  And why sometimes we need them and sometimes we don't? 

I've got to put these things together 90 different ways before I stumble across the correct combination of which to use and when, and many times I honestly don't know WHY it works one way and not the other.   Somebody needs to give us a nice tutorial or set of examples explaining all those different little tweaks.
http://bit.ly/TextImage -- Library of QB64 code to manipulate text and images, as a BM library.

http://bit.ly/Color32 -- A set of color CONST for use in 32 bit mode, as a BI library.

DarthWho

  • Hero Member
  • *****
  • Posts: 4039
  • Timelord of the Sith
Re: Confusion with DLL output (Programmer Error)
« Reply #3 on: May 04, 2014, 05:44:17 pm »
well here's a quick tutorial:
DECLARE LIBRARY is just in general for static and shared object libraries
Use DECLARE DYNAMIC LIBRARY for any DLLs that aren't built into QB64
As far as I'm aware DECLARE CUSTOMTYPE LIBRARY allows you to use static libraries that reference DLLs.
DECLARE STATIC LIBRARY prioritizes static libraries over shared object libraries but on windows is effectively Identical to vanilla DECLARE LIBRARY

Rule for Byval:
Use whenever you want to pass the value of the input instead of a pointer to the input.
FastMath 1.1.0 released: http://dl.dropbox.com/u/12359848/fastmath.h

BTC: 1DGmy7rBZ15Y1nFJXkoE8BkvmMu6DxSMM4
LTC: LRNzAapRvQEuuEGwuLTG1f6nuHaf7tqkn7

Johny B.

  • QB64 Partner Site Owner
  • Hero Member
  • *
  • Posts: 1497
Re: Confusion with DLL output (Programmer Error)
« Reply #4 on: May 05, 2014, 03:14:13 am »
I admit that it can all get pretty confusing, between the pointers and the data types and the differ DECLARE * LIBRARY forms. However, to quote Paul Simon, "The answer is easy if you take it logically". It's worth pointing out that I use Linux, so my understanding of the specifics of DLLs is sketchy.

First off, the different types of DECLARE LIBRARY. http://www.qb64.net/forum/index.php?topic=2091.0 is a good read; I have summarised the information here.

DECLARE LIBRARY: Tells QB64 that the functions we are going to declare are already declared somewhere in QB64. This would be
case for most standard C functions, and any SDL functions that are already defined.

DECLARE LIBRARY "" (notice the empty quotes): Tells QB64 that the library is already included in QB64 somehow (just like above), but it is not declared anywhere, so a declaration needs to be created. Choosing between this option and the one above is really just a guessing game; try the one above first.

DECLARE LIBRARY "foo": This is where we indicate we want to use an external file. You should not provide an extension. QB64 will search for "foo.lib", then "foo.a", then "foo.o", and use the first one it finds. It searches in the current directory, then /usr/lib (on Linux) and I presume system32 on Windows (note that this behaviour is broken on Linux systems which store libraries in /usr/lib/i386-linux-gnu etc., and you might have to give an absolute path) It may search in more places, but I'm not sure. Also, on Linux, the name is prefixed with "lib", so it searches for "libfoo.a" etc. In addition to this, QB64 also searches for "foo.h" in the current directory, and in other systemwide placed (/usr/include on Linux, don't know about Windows). The idea here is that you have the foo library, and foo.h which gives declarations for the functions in foo. However, it is also a convenient way of inserting C functions into your program without pre-compiling them; QB64 will not mind if there is a .h file and no compiled library to go with it. (Linux: it seems to pick up .so libraries as well, in preference to a .a file.) I personally think it should allow "file.cpp" as well, but hey, it doesn't.

Detour: What are .lib, .a and .o files?
.lib files seem to be a Windows-only library type that are mainly used to accompany .dll files. From what I read, the .lib contains 'stubs' that fill in the calls to the dll when you link it. When the program is actually run, the dll is loaded and some runtime magic happens.
.o files are object files. This is what you get when you compile a single file with g++. It can contain multiple functions, and when you create your final program, this file is linked in with all the other .o files and any libraries to produce a final product.
.a files are archives, and are essentially a collection of .o files. They've just been packaged together to make things neater.

DECLARE DYNAMIC LIBRARY "foo": Sometimes, you want to link against a .dll or .so file directly (.so is a shared library on Linux). Since the linking process for these is different, the command is different too. It's important to keep in mind that there is no type checking, so the compiler will let you declare the function however you want. If you get it wrong, a crash is the best case scenario. Worst case, you get valid but wrong results, and you don't notice until much later. Note that QB64 will complain if there is a .h file with the same name. Also, keep in mind that the library needs to be present at runtime.

DECLARE STATIC LIBRARY "foo": Seems to the same as regular DECLARE LIBRARY "foo", but the wiki says it prefers .lib, .a and .o over .so files if there is one with the same name.

DECLARE CUSTOMTYPE LIBRARY "foo": Much like regular DECLARE LIBRARY, but will aggressively type-cast parameters (hence the "custom types"). Useful for when you're lost in a sea of typedef's and the compiler is complaining, but care should be taken to ensure the data sizes do match.


Okay, now we've got that out of the way, let's look at data types and pointers. First off, let's have a C function for study. This one will do nicely:
Code: [Select]
float cadd(float a, float b) {
  return a + b;
}
Save that in a file called caddfile.h, and pop the following into QB64:
Code: [Select]
DECLARE LIBRARY "caddfile"
FUNCTION cadd! (BYVAL x!, BYVAL y!)
END DECLARE

PRINT cadd!(34.2, 11.19)
Compile that, and if the stars are aligned, you should be informed about the sum of those two numbers. Now let's look closer. In the C header (actually, your code is always compiled as C++. There are differences between the two), we've declared a function that takes two arguments, each with the type float, and returns a float (this is a C++ float, not a QB64 _FLOAT, of course). Simple enough.

Now, if you look at the BASIC, you'll see the function declaration is cadd! (BYVAL x!, BYVAL y!), which means this function takes two SINGLE arguments, both passed BYVAL, and returns a SINGLE. At this point, you may be asking why it was a float before and a SINGLE now. If we refer to this handy little chart I put together:
Code: [Select]
C type      QB name               QB Symbol
NA        _UNSIGNED _BIT        ~`
NA        _BIT                  `
NA        _BIT * n              `n
NA        _UNSIGNED _BIT * n    ~`n
int8_t      _BYTE                 %%
uint8_t     _UNSIGNED _BYTE       ~%%
int16_t     INTEGER               %
uint16_t    _UNSIGNED INTEGER     ~%
int32_t     LONG                  &
uint32_t    _UNSIGNED LONG        ~&
int64_t     _INTEGER64            &&
uint64_t    _UNSIGNED _INTEGER64  ~&&
float       SINGLE                !
double      DOUBLE                #
long double _FLOAT                ##
offset          void*       _OFFSET               %&
We can see that the QB SINGLE is actually a C++ float under the hood. Any time you want to pass a value to a C++ routine, make sure you look up the type in that table. The one type that isn't there is a C++ int. It's usually a LONG, but can (in theory) be different on different machines/OSes.

What's that you say? I haven't explained BYVAL? Very well. Buckle up. This could get rough, depending on how good your pointer work is.

Just like in the original QBasic, values are passed to external routines 'by reference'. This means that, if you were to not use BYVAL at all, calling cadd!(23, 53) would NOT pass 23 and 53 as arguments. Instead, it stores those values in memory, say at addresses &H80BF0000 and &H80BF0004. It is these address that are then passed to the routine. This is what is meant by passing a 'pointer to the value' - we pass a value that tells us where the value we want actually is.

As weird as this may be, it's just how QB SUBs behave. Consider calling SUB bar as "bar x, y". bar is free to change x and y, and this change will be reflected in the main program. Similarly, passing a pointer to a variable (remember, this means "the address of the variable") allows the C++ function to change the data in that variable. It's important to understand that you don't use _OFFSET at all here. The compiler takes care of all the dirty work involved in calculating the variable's address.

BYVAL is a way of telling the compiler, "No, don't pass an address. Pass the value itself." Now, when you use BYVAL in the declaration of cadd, the compiler passes the value itself. If you pass a variable, the program reads that variable, and passes the value contained in it. The cadd function has no knowledge of where its arguments came from, and any changes to them do not affect the caller.

In practical terms, the rule is: By default, use BYVAL. If the C declaration for that argument has a * between the type and the name, it indicates that that variable should be passed 'by reference', so you don't use BYVAL. If you're dealing with a string, read on.

Strings:
Strings are a separate kind of beast to numeric values. When we pass a string, we never pass the string itself. We always pass a pointer to the start of the string. This means that you never use BYVAL. In C, a string will either be declared as char * or char[], which are basically the same thing for our purposes. Let's consider a concat function, which joins two strings and returns the result:
Code: [Select]
DECLARE LIBRARY "concat"
FUNCTION concat$(s1$, s2$)
END FUNCTION
We then might call it like this: PRINT concat$("Hello " + CHR$(0), "World" + CHR$(0)). Notice that we must finish each string passed with CHR$(0). In C, this signifies the end of a string, and we must add it explicitly. That will work reasonably well, but there's a catch. If concat() is dynamically allocating memory for the return string with malloc() (which it probably is) then repeated calls will leak memory. This is described in extreme detail here, http://www.qb64.net/forum/index.php?topic=12080.0 but the bottom line is just be aware that this leak exists, and you can probably ignore it outside of tight loops.

_OFFSET:
I'd like to say a bit about the _OFFSET type. This type is basically there for when you have to shunt around a pointer between various library calls. Often an initialisation function will return a pointer, which holds some data that the library need to function properly. Successive calls to other functions then require this pointer to be passed to them, so they can access/change the data as they need to. In this case, the first function is declared FUNCTION init%& (), and other functions will have a BYVAL data_struct%&. It may be surprising that it's passed BYVAL, but consider what you're actually passing: the function wants the address of the data, not the data itself.

See, wasn't that easy?

If you don't have the will to read through all that, here's the takeaway message regarding datatypes:

- Make sure you match the QB64 datatype with its C++ equivalent. Use the table above to help you with that, and remember int is usually a LONG (&).
- Use BYVAL usually, but if the C code takes a pointer (designated by the * symbol, as "int *x"), don't use BYVAL.
- Strings are never passed BYVAL, but make sure you add a CHR$(0) when you pass them to C code, otherwise it can't tell how long it is, and problems will arise (some functions will get you to pass a separate parameter with the length though. Read the documentation).
- _OFFSET is mainly needed when you're passing around pointers to data structures (I have used on other odd occasions, but I can't remember off the top of my head when).

That's a lot of stuff to read through. It makes sense to me, but it could very well be incomprehensible to everyone else, and it's entirely possible that there are factual errors. I invite specific questions, and recommendations for improving this text.
« Last Edit: November 29, 2016, 11:38:18 pm by Johny B. »

  • Print