All Classes Namespaces Files Functions Variables Typedefs Enumerations Friends Macros Pages
Classes | Namespaces | Macros | Functions
mapped_file.hpp File Reference

Interface for using mmaped files with threads. More...

#include "paal/utils/type_functions.hpp"
#include "paal/utils/irange.hpp"
#include "paal/data_structures/thread_pool.hpp"
#include <boost/iostreams/device/mapped_file.hpp>
#include <string>
#include <vector>
#include <thread>

Go to the source code of this file.


class  paal::data_structures::mapped_file
 data structure that gets new lines for many threads More...


 global namespace of project.
 Data Structure namespace.


template<typename Functor >
auto paal::data_structures::for_each_line (Functor f, std::string const &file_path, unsigned threads_count=std::thread::hardware_concurrency())
 for_every_line function provides basic functionality for processing text files quickly and clearly. Thanks to mmap() functionality it doesn't have to seek through file but it loads it to virtual memory instantly and uses only ram cache to do that. Furthermore file is split instantly - thanks to that it can be processed effectively using threads. Downside of using mmap is that this functionality will not work effectively if threads have small jobs to be done comparing reading the line charge. It's supposed to work with O(threads_count) memory usage but remember - RES (resident size) stands for how much memory of this process is loaded in physical memory, so file pages loaded in ram cache are added to that value. More...

Detailed Description

Interface for using mmaped files with threads.

Karol Wegrzycki

Definition in file mapped_file.hpp.