kirit.com

Created 26th April, 2005 13:12 (UTC), last edited 22nd December, 2016 03:24 (UTC)

Writing about C++, Programming, Fost 5, the web, Thailand and anything else that catches my attention—with some photos thrown in

Fost 5 release 5.18.06.45062 now out

Posted 24th June, 2018 03:08 (UTC), last edited 24th June, 2018 10:35 (UTC)

This is the last release that will use Boost build (bjam). We're joining the cmake bandwagon instead. This is more of a sideways step than a forwards one, but hopefully might make some things easier in the long run, although in the short term there are likely poor choices in how to structure the builds.

We are deleting the old boost build files (Jamroots and Jamfiles) from everywhere and tidying up a few other small things at the same time.

With cmake it should become simpler to compile using odd compilers (especially cross compiling) and its ability to generate Visual Studio and XCode project files should make it simpler to work on those platforms, not to mention all of the IDEs that use cmake natively.

Some early advantages are that project layouts are now much more flexible because the cmake link directives don't need paths to targets and also that openssl can be easily replaced by boringssl.

As part of this work we're now dropping fost-orm and fost-py. The only library from fost-orm carried forwards is fost-jsondb which is now part of fost-beanbag. There are no new Python bindings.

Also, due to changes in OpenSSL the oldest version of Boost that will be supported by the cmake based builds will be 1.62.0.

Building on Linux & Mac

git clone --branch=5.18.06.45062 --recursive git@github.com:KayEss/fost-hello.git
cd fost-hello
Boost/build
hello/compile
dist/bin/hello-world-d

On the Mac you will need to set DYLD_LIBRARY_PATH before running hello-world-d

export DYLD_LIBRARY_PATH=dist/lib
dist/bin/hello-world-d

Download locations

Applications

  • beanbag — Stand alone transactional JSON database server — git@github.com:KayEss/beanbag.git
  • beanbag-seed — Seed project for giving you a starting point to develop web applications using Beanbag — git@github.com:KayEss/beanbag-seed.git
  • fost-hello — Sample seed project — git@github.com:KayEss/fost-hello.git
  • mengmon — Stand alone web server — git@github.com:KayEss/mengmom.git
  • wright — Experimental build system — git@github.com:KayEss/wright.git

Libraries

  • f5-cord — First version of a new string library with compile time string and Unicode support — git@github.com:KayEss/f5-cord.git
  • f5-threading — Preview of the first Fost 5 library which includes help for threading — git@github.com:KayEss/f5-threading.git
  • fost-aws — Amazon AWS and OpenStack — git@github.com:KayEss/fost-aws.git
  • fost-android — Eclipse project for Android that allows Fost 4 and Beanbags to be used on mobile devices — git@github.com:KayEss/fost-android.git
  • fost-android-ndk — The native code for Android. Includes required parts of Boost configured to use the standard Android build system.
  • fost-beanbag — Transactional JSON database — git@github.com:KayEss/fost-beanbag.git
  • fost-base — Build system and core libraries — git@github.com:KayEss/fost-base.git
  • fost-internet — Internet protocols, servers & clients — git@github.com:KayEss/fost-internet.git
  • fost-meta — All libraries in one wrapper — git@github.com:KayEss/fost-meta.git
  • fost-orm — Object/Relational mapping — git@github.com:KayEss/fost-orm.git
  • fost-postgres — PostgreSQL — git@github.com:KayEss/fost-postgres.git
  • fost-py — Python (2.x) bindings — git@github.com:KayEss/fost-py.git
  • fost-web — Web server libraries — git@github.com:KayEss/fost-web.git
  • fost-wright — Experiment in a build system — git@github.com:KayEss/fost-wright.git

Detailed change log

cord

  • Add a memory method to the u8view so that the underlying memory buffer can be accessed.

fostgres

  • Add support for password configuration on the connection strings.
  • Allow database connection string settings to come from the environment.

fost-base

  • Add coercions for jcursor index value types.
  • Add some basic base16 support.
  • Add various base32 encoding flavours (base32, base32hex and base32c)
  • Add a fork of Crypto++'s fork of tweetnacl to generate ed25519 signing key pairs.

fost-internet

  • Add fost-hod for handling packet oriented data over TCP.

fost-web

  • Add a pathname view and a file serving view.

Categories:

Fost 5 release 5.18.03.45059 now out

Posted 26th March, 2018 04:11 (UTC), last edited 26th March, 2018 08:53 (UTC)

The Mac builds are finally back! This means we now have Linux, Mac and Android. Regaining iOS support is still something for the future.

The Mac build does drop the Python 2 bindings and the Wright application. Wright isn't going to be a loss, and the Python 2 bindings need to be deprecated in favour of a newer set of Python 3 ones.

One upshot of this is that the oldest compiler we need to support is now likely to be the Mac one rather than the Android one, but it's hard to tell because Apple have a completely different versioning scheme than the llvm project uses.

There have been many changes to the string handling. This is still leading up to the large refactoring. The u8view has become much more useful and there is a shared version of this in the works. Eventually this will lead to the replacement and deprecation of the fostlib::string for a variant based constant string, and the use of std::string everywhere else.

There has also been more cleaning up and removing of Boost libraries. You may find you need to now include Boost threading headers or Boost function headers.

Building on Linux & Mac

git clone --branch=5.18.03.45059 --recursive git@github.com:KayEss/fost-hello.git
cd fost-hello
Boost/build
hello/compile
dist/bin/hello-world-d

On the Mac you will need to set DYLD_LIBRARY_PATH before running hello-world-d

export DYLD_LIBRARY_PATH=dist/lib
dist/bin/hello-world-d

Download locations

Applications

  • beanbag — Stand alone transactional JSON database server — git@github.com:KayEss/beanbag.git
  • beanbag-seed — Seed project for giving you a starting point to develop web applications using Beanbag — git@github.com:KayEss/beanbag-seed.git
  • fost-hello — Sample seed project — git@github.com:KayEss/fost-hello.git
  • mengmon — Stand alone web server — git@github.com:KayEss/mengmom.git
  • wright — Experimental build system — git@github.com:KayEss/wright.git

Libraries

  • f5-cord — First version of a new string library with compile time string and Unicode support — git@github.com:KayEss/f5-cord.git
  • f5-threading — Preview of the first Fost 5 library which includes help for threading — git@github.com:KayEss/f5-threading.git
  • fost-aws — Amazon AWS and OpenStack — git@github.com:KayEss/fost-aws.git
  • fost-android — Eclipse project for Android that allows Fost 4 and Beanbags to be used on mobile devices — git@github.com:KayEss/fost-android.git
  • fost-android-ndk — The native code for Android. Includes required parts of Boost configured to use the standard Android build system.
  • fost-beanbag — Transactional JSON database — git@github.com:KayEss/fost-beanbag.git
  • fost-base — Build system and core libraries — git@github.com:KayEss/fost-base.git
  • fost-internet — Internet protocols, servers & clients — git@github.com:KayEss/fost-internet.git
  • fost-meta — All libraries in one wrapper — git@github.com:KayEss/fost-meta.git
  • fost-orm — Object/Relational mapping — git@github.com:KayEss/fost-orm.git
  • fost-postgres — PostgreSQL — git@github.com:KayEss/fost-postgres.git
  • fost-py — Python (2.x) bindings — git@github.com:KayEss/fost-py.git
  • fost-web — Web server libraries — git@github.com:KayEss/fost-web.git
  • fost-wright — Experiment in a build system — git@github.com:KayEss/fost-wright.git

Detailed change log

cord

  • Add empty to the u8view and buffers.
  • Add ability to output u8view and move overloaded operator <<s to std namespace.
  • Allow appending of a u8view to a std::string.
  • u8views can be concatenated to form std::strings.
  • u8view supports operator <.
  • lstring now also has a string literal, _l.
  • Add empty() member to lstirng.
  • The const shared string buffers need to be immutable.
  • u8view can be explicitly converted to a std::string.
  • u8view is now comparable with const char *.

threading

  • Don't include any Fost headers.
  • Bypass the coroutine unwind exceptions for the reactor and sync.
  • Move the reactor_pool to reactor.hpp
  • The reactor_pool can be closed and its threads joined at any time.

fostgres

  • Fix the fg parser to allow whitespace and comments in more places.

fost-aws

  • Removed use of boost::function.

fost-base

  • Fix bug to handle requirements for autotest libraries properly.
  • Removed Boost lambda and bind
  • Remove previously deprecated fostlib::counter.
  • Allow the C++ version to be specified as std-version in the requirments.jam file.
  • crypto_compare works with Unicode strings.
  • More tagged strings can now be coerced to json and to f5::u8view.
  • Improve the converting nullable constructor so it uses explicit type conversions.
  • Make more string utilties compatible with f5::u8view, at the cost of wide character literals.
  • Make nullable<f5::u8view> coercions from JSON not throw errors if the value isn't a string.
  • Improve the jcursor APIs to better support u8view and move semantics.
  • Add f5::lstring as an option for storage inside fostlib::json.
  • Control characters need to be escaped when producing JSON strings.
  • Can now mint a JWT with an initial payload.
  • Remove uses of boost::function and make worker const sane.
  • Add missing namespaces to the log definition macros.
  • Small readability improvement to the default stdout logger.
  • Crypto hashes now take a f5::u8view which should remove some memory allocations.
  • Deprecate fostlib::utf::u8_view in favour of f5::u8view
  • Move utf::u8_view to f5-cord.
  • Remove array_view and replace with f5::array_view from f5-cord.
  • Deprecate use of the fostlib::string JSON coercion in favour of utf::u8_view.

fost-internet

  • Make the query_string API a bit less pathalogical by caching the stringified version.
  • JSON arrays used as header sub-values are now set as the JSON string.
  • Removed all uses of boost::function

fost-orm

  • Remove all instances of boost::function

fost-postgres

  • Put session variable names in double quotes.
  • Changed the upsert function to also take an array of strings for a RETURNING clause.

fost-py

  • Fix a problem with Boost 1.65

wright

  • Work is distributed between workers more evenly.

Categories:

Fost 5 release 5.17.12.45058 now out

Posted 28th December, 2017 04:42 (UTC), last edited 28th December, 2017 08:32 (UTC)

This release includes a step towards revised string handling. The eventual goal is to be able to remove the fostlib::string from most places in favour of a smarter structure based on a std::variant that can better handle different string sources. This work is ongoing and you can see progress in the develop branches.

We're also taking a look at macos and Windows again, but both platforms are quite difficult so no real progress yet.

Building on Linux

You will need a C++14 compiler. Recent versions of either gcc or clang are suitable.

git clone --branch=5.17.12.45058 --recursive git@github.com:KayEss/fost-hello.git
cd fost-hello
Boost/build
hello/compile
dist/bin/hello-world-d

Download locations

Applications

  • beanbag — Stand alone transactional JSON database server — git@github.com:KayEss/beanbag.git
  • beanbag-seed — Seed project for giving you a starting point to develop web applications using Beanbag — git@github.com:KayEss/beanbag-seed.git
  • fost-hello — Sample seed project — git@github.com:KayEss/fost-hello.git
  • mengmon — Stand alone web server — git@github.com:KayEss/mengmom.git
  • wright — Experimental build system — git@github.com:KayEss/wright.git

Libraries

  • f5-cord — First version of a new string library with compile time string and Unicode support — git@github.com:KayEss/f5-cord.git
  • f5-threading — Preview of the first Fost 5 library which includes help for threading — git@github.com:KayEss/f5-threading.git
  • fost-aws — Amazon AWS and OpenStack — git@github.com:KayEss/fost-aws.git
  • fost-android — Eclipse project for Android that allows Fost 4 and Beanbags to be used on mobile devices — git@github.com:KayEss/fost-android.git
  • fost-android-ndk — The native code for Android. Includes required parts of Boost configured to use the standard Android build system.
  • fost-beanbag — Transactional JSON database — git@github.com:KayEss/fost-beanbag.git
  • fost-base — Build system and core libraries — git@github.com:KayEss/fost-base.git
  • fost-internet — Internet protocols, servers & clients — git@github.com:KayEss/fost-internet.git
  • fost-meta — All libraries in one wrapper — git@github.com:KayEss/fost-meta.git
  • fost-orm — Object/Relational mapping — git@github.com:KayEss/fost-orm.git
  • fost-postgres — PostgreSQL — git@github.com:KayEss/fost-postgres.git
  • fost-py — Python (2.x) bindings — git@github.com:KayEss/fost-py.git
  • fost-web — Web server libraries — git@github.com:KayEss/fost-web.git
  • fost-wright — Experiment in a build system — git@github.com:KayEss/fost-wright.git

Detailed change log

cord

  • Add string equality comparison and substr operations to u8view.
  • Fill in more of the lstring comparison members.
  • Add shared buffers who can own strings (and other things).
  • Rename array_view to buffer as it is not always immutable.
  • Add u8view and UTF8 decode function.
  • Added an array_view with some constexpr support.

fostgres

  • Add scripting call setting for manipulating settings.
  • Add parser support for all JSON atom values.
  • Add RETURNING support to PUT request for objects.
  • Move some tests so Postgres is not needed for a normal build.

Categories:

Pacific++ trip report

Posted 31st October, 2017 02:25 (UTC), last edited 2nd November, 2017 01:31 (UTC)

Pacific++ is the first programming language conference I've been to, and I have to say that actually being there is a big step up from just watching videos on youtube. The conference was small and intimate, which was great. I think I probably managed to talk to nearly half of the about 70 attendees. We got mugs

Day 1

Chandler Carruth opened with a live demo (video) of how to build clang and it's tooling and then went on to demo some of the other features.

I've been building a toolchain for our internal use so that we're able to make use of the new coroutines features that shipped with clang 5. I'd kind of expected this to be a short term thing so hadn't put too much effort into it, so it was interesting to learn that Google actually do this weekly just using HEAD of master for all of their builds. Maybe tidying this up for long term use will be good.

One thing Chandler didn't talk about is how they distrubute the toolchain to developers, a problem I think we've solved quite well using docker. I think it's something that I should probably blog about.

Making use of the llvm linker (-fuse-ld=lld, it's much faster) and the use of the link time optimiser (-flto=thin) also looked worthwhile.

He also showed us the CFI sanitizer (-fsanitize=cfi) which is designed to help harden your code in production (important for our networking code).

Chandler only needed four and half minutes to compile all of clang, but he did later admit that he was using a 128 core monster he has at home built especially to make clang builds fast.

Next up Toby Allsopp talked about coroutines (video). He gave a general high level overview of how they work and some of the things you can do with them, and most importantly, showed the benefits they bring in terms of code clarity. He has some great diagrams of the control flow through the coroutine and its calling function which would certainly help my tutorial articles. I'll be linking his from the introduction article once it's up on youtube.

Matt Bentley took us through his higher performance list implementation (video) which uses an allocation scheme that tries to keep list cells contiguous in memory as far as possible. It turns out that there are many places that the common wisdom of always just using a vector isn't the best for performance and he showed us places where list implementations (especially his) do win out.

Dean Michael Berris explained how XRay works, which is a tool for gathering performance data about the execution of your code, and is fast enough that it can be used in production systems with the data gathering turned on and off whilst the system is running.

This looks like a very interesting capability that is now included in newer versions of clang (and is yet another reason why building your toolchain is worthwhile).

Dean took us through the low level implementation details of how the instrumentation is turned on and off at runtime and why the overhead is so low. He also showed some of the (offline) analysis tools that come with XRay.

Right now it doesn't support shared objects (dynamic linking with .so files), but Dean promised me that was work in progress. Still more reason to stay as near to HEAD with llvm as possible—and, if whoever is working on it needs somebody to test it out then please let me know :)

The final talk was Christian Blume's about his transwarp library, which is able to build up a graph (acyclic tree only) of tasks with the dependencies and then execute it multiple times (the other approach is to build the tree for each group of tasks and then rip it down when they've executed).

It takes a very different approach to how this would look with coroutines or the ranges TS, which may be easier to work with where the dependancy graph is only known at runtime.

At the end of the time there was much lively discussion in the bar followed by the speaker's dinner which I'd decided to pay the extra for given that I was there on my own and figured it would be a good opportunity to talk to more people (spoiler, it was well worth it).

At the dinner Chandler came out with a very interesting rant on the difficulties compilers have with user defined literals, especially for compile string handling of the type I've been playing around with in f5-cord's tstringclass — Oops, and somethingh I'll have to think more about. Not only are there parsing problems with them, but more importantly, compile time substring processing is especially costly in terms of compile times (thankfully not something I care about).

Day 2

The second day started off with Jason Turner (video) telling us about the importance of noexcept. I'd always not worried too much about it because I thought that the overhead of having to manage the terminate condition mandated in the standard would negate any possible code generation improvements. Jason managed to show that was categorically false, and even more importantly demonstrated many code improvements directly driven by trying to make things noexcept that should be.

Carl Cook showed a number of techniques used by high frequency trading developers to reduce latency when trading. I've been watching a lot of high throughput videos by games developers and then applying those techniques to Postgres database designs, so it was interesting to see some low latency examples too. For some of the things we want to do latency can be important so now I need to think about how to apply these techniques to databases.

Wanting to implement state machines was a big reason for me to pick up coroutines, after watching Dominic Robinson's ACCU talk, so it was great to see what Nick Sarten had done through using std::variant. He benchmarked a few different approaches, including one based on inheritance. This implementation, like a coroutine based one, performs a memory allocation on state change and the benchmarks showed just how expensive that would be. Something we're going to need to think a lot more about.

Sarah Smith is a much braver person than me clearly. I've done some live coding talks, but she built an entire application using Qt and then ran it on her desktop and phone (iOS, of course the Android had problems connecting to the dev environment, as they always do).

She also talked about her history and how she ended up a mobile developer and entrepreneur. I always love to hear stories about how people ended up where they are.

The final talk was Tom Isaacson showing how, what look like the same warnings, actually differ a lot between compilers and how that affects cross platform development. I don't switch warnings to errors because of the range of different compilers we use, but I can certainly see the appeal and it would be good to tighten this up and turn on more warnings. He showed some interesting ones that are not turned on by default.

Go to a conference

This was my first C++ conference (actually, my first real tech conference), and it was well worth the trip. I've watched a lot of videos from other conferences, and although great, it's not anything like actually being there and getting to talk to people.

I think my next mission should be to make it to the ACCU conference in the UK.


I'll link the videos as they come online. If any other links are wrong, or could be better elsewhere please let me know. I've probably misunderstood what people were trying to say in their talks, mistakes are my own.


Categories:

Fost 5 release 5.17.09.45051 now out

Posted 23rd September, 2017 03:51 (UTC), last edited 23rd September, 2017 04:12 (UTC)

The layout changes to fostlib::json have now landed. Previously the atomic parts of JSON (bool, int, double, string etc.) where stored in a separate variant to the one that handled the atomic, array and object parts. By bringing this all into one variant the type is smaller and dealing with atomic types is faster.

As part of this change we also changed the object and array to being stored by a shared_ptr rather than directly. This was always the intended design, and means that now taking a copy of a json is significantly cheaper. We have some more changes coming now that we have this base that should make the type smaller still and further improve performance.

We're also still moving towards improving string handling with a few new immutable string types in the pipeline and a move towards using immutable string views in many more places.

It's also about time that we get serious with Python 3 support. All of our projects are now using this and we need to be able to support bindings for it correctly. There's still some debate about whether that should be a branch in fost-py or a separate library. In any case the entirety of the Python 2 version will be replaced with something completely new.

Building on Linux

You will need a C++14 compiler. Recent versions of either gcc or clang are suitable.

git clone --branch=5.17.09.45051 --recursive git@github.com:KayEss/fost-hello.git
cd fost-hello
Boost/build
hello/compile
dist/bin/hello-world-d

Download locations

Applications

  • beanbag — Stand alone transactional JSON database server — git@github.com:KayEss/beanbag.git
  • beanbag-seed — Seed project for giving you a starting point to develop web applications using Beanbag — git@github.com:KayEss/beanbag-seed.git
  • fost-hello — Sample seed project — git@github.com:KayEss/fost-hello.git
  • mengmon — Stand alone web server — git@github.com:KayEss/mengmom.git
  • wright — Experimental build system — git@github.com:KayEss/wright.git

Libraries

  • f5-cord — First version of a new string library with compile time string and Unicode support — git@github.com:KayEss/f5-cord.git
  • f5-threading — Preview of the first Fost 5 library which includes help for threading — git@github.com:KayEss/f5-threading.git
  • fost-aws — Amazon AWS and OpenStack — git@github.com:KayEss/fost-aws.git
  • fost-android — Eclipse project for Android that allows Fost 4 and Beanbags to be used on mobile devices — git@github.com:KayEss/fost-android.git
  • fost-android-ndk — The native code for Android. Includes required parts of Boost configured to use the standard Android build system.
  • fost-beanbag — Transactional JSON database — git@github.com:KayEss/fost-beanbag.git
  • fost-base — Build system and core libraries — git@github.com:KayEss/fost-base.git
  • fost-internet — Internet protocols, servers & clients — git@github.com:KayEss/fost-internet.git
  • fost-meta — All libraries in one wrapper — git@github.com:KayEss/fost-meta.git
  • fost-orm — Object/Relational mapping — git@github.com:KayEss/fost-orm.git
  • fost-postgres — PostgreSQL — git@github.com:KayEss/fost-postgres.git
  • fost-py — Python (2.x) bindings — git@github.com:KayEss/fost-py.git
  • fost-web — Web server libraries — git@github.com:KayEss/fost-web.git
  • fost-wright — Experiment in a build system — git@github.com:KayEss/fost-wright.git

Detailed change log

cord

  • Add a _t literal for creating tstring instances. This allows us to deprecate the vstring template.
  • Added iostream output for lstring.

fostgres

  • Add -h and -U options to fostgres-test to control how to access the database.

fost-base

  • Fix generation of log messages from JSON where the modules are an array.
  • Count the number of times an exception has been absorbed.
  • Remove variant for storing atoms in json
  • Don't print empty objects in the log stream.
  • Load the global logging configuration much later so it can be changed using the -i or -j command line switches.
  • Better integrate utf::u8_view and string.
  • Make array_view sliceable.
  • There is now an explicit constructor from std::string for the u8_view.
  • Make dates fully comparable.
  • Improve error report when coercing JSON to strings fails.
  • The unexpected_eof error can now take a boost::system::error_code for use with socket errors.

fost-internet

  • Ampersands, single quote and dollar symbol added to allowed URL path characters.
  • The conversion of HTTP request file paths to strings now works for a wider range of inputs. The conversion to boost::filesystem::wpath is problematic with Boost versions prior to 1.64 due to the way they use codecvt.

fost-orm

  • Improve error report when a beanbag update precondition wasn't met.

fost-postgres

  • Make it possible to iterate over the columns in the row.
  • Change libpqxx paths for latest code.
  • Implement assignment and pre-increment for recordset::const_iterator

fost-py

  • The spider always outputs a test.txt file with all results in it.

fost-web

  • The template middleware now uses the headers from the wrapped view instead of throwing them away.
  • Logging middleware also records the processing duration.

Categories: