Embedded Guru

Like functions, we can also pass variable arguments to macros. Similar to functions, we need to include ellipses (...) in macro definition. __VA_ARGS__ is a varadic macro. Declaration syntax is similar to varadic macro. A sequence of three full stops "..." is used to indicate that one or more arguments can be passed. During macro expansion each occurrence of __VA_ARGS__ in the macro replacement list is replaced by the passed arguments. Example: If we want to define our own printf() function, which also prints the file and line number. // Our implemented function void realdbgprintf (const char *SourceFilename,                     int SourceLineno,                     const char *CFormatString,                     ...); #define dbgprintf(...) realdbgprintf (__FILE__, __LINE__, __VA_ARGS__) dbgprintf ("Hello, world"); ex...

C allows you to define a function which can accept a variable number of arguments. To use such functionality, you need to include stdarg.h header file. This file provides the functions and macros to implement the functionality of variable arguments. Important Points: Syntax of using variable length arguments:         return_type function_name(parameter_list, int num, ...); Last argument as elipses i.e. (three dots ...) Argument before elipses in the function is always 'int' which represents the number of arguments passed Create va_list type variable in the function definition. This is defined in stdarg.h file Initialize va_list variable with the argument passed using va_start macro and the first parameter Use va_arg macro to access each item inside the va_list Clean the memory assigned to va_list using va_end macro va_start function contains the magic code to initialize the va_list with the correct stack pointer. It must be passed the last n...

License of an upstream may change in future. LIC_FILES_CHKSUM is used to avoid these changes getting unnoticed. In do_populate_lic stage, if checksums do not match, the build will fail. To track the Licenses, the syntax is  LIC_FILES_CHKSUM = "file://COPYING;md5=xxxx" If your source code has the license information at the beginning, you can use beginline, endline: LIC_FILES_CHKSUM = "file://hello.c; beginline=5;endline=10;md5=xxxx" Let's understand by the sample hello world recipe present in poky/meta-skeleton/recipes-kernel/hello-mod/files COPYING is the file here stores the license information for all the source files. To calculate md5 of the file, use 'md5sum filename' command. You can see from the above md5sum output and md5 value present in LIC_FILES_CHKSUM is same. Let's see what happens if I modify 'COPYING' file and don't update the md5 value in LIC_FILES_CHKSUM filed of the recipe.

We can find out what is the first kernel function called when the user space code gets into the kernel using 'max_graph_depth' file. This file can only be used with the function tracer. max_graph_depth file contains the maximum depth it will trace into a function. Default value is 0, which means unlimited. If you 'echo 1 > max_graph_depth', it will only the trace the first function and ignore the other functions. You can see, 'do_syscall_64()' is the function through which the user space code gets into the kernel space. If we want to find out the next function after 'do_syscall_64()', 'echo 2 > max_graph_depth'

The function graph tracer also traces the interrupts. You can tell the function graph tracer to disable tracing of interrupts by writing 0 in 'options/funcgraph-irqs' file. $ echo 0 > options/funcgraph-irqs The below screenshot is before disabling interrupts. After I disable interrupts, there is no tracing information related to interrupts (e.g. smp_apic_timer_interrupt).

'trace_options' file allows user to control the level of information displayed when we do cat on 'trace' or 'trace_file' You can see from the above screenshot, There are 26 options available for the user to control. There are some tracer specific options which are not listed in this file and are present in 'options' directory. The words which begin with "no" represents trace options that are disabled. To enable an option. $ echo 'sym-offset' > trace_options To disable an option, add "no" $ echo 'nosym-offset' > trace_options. Let's try some of the options. 1. sym-offset: Enabling this option, you will see function name plus offset in the function. By default, this is disabled. 2. print-parent: This option displays the calling function. By default, this option is disabled. 3. function-fork: This option enables tracing the children when set_ftrace_pid is set. 4. irq-info: ...

In Linux, Serial Port input can be configured in two ways: 1. Canonical Mode 2. Non-Canonical Mode Canonical Mode : In canonical mode, read on the serial port will not return until a new line, EOF or EOL character is received  by the Linux Kernel. In this more, read will always return an entire line, no matter how many bytes are requested. Non-Canonical Mode: In non-canonical mode, read on the serial port will not wait until the new line. Number of bytes read depends on the MIN and TIME settings. MIN: Minimum number of bytes that must be available in the input queue in order for read to return TIME: How long to wait for input before returning, in units of 0.1 seconds. MIN and TIME Behavior MIN = 0, TIME = 0 Read returns immediately with as many characters available in the queue, up to the number requested. If no characters in the queue, read returns 0 MIN = 0, TIME > 0 Read waits ...

What is Baud Rate? Baud rate is the number of signal units transmitted per unit time, this is different from Bit rate, which is the number of bits transmitted per unit time. Difference between bit rate and baud rate is that one signal unit can carry one or more than one bit of information. Bit rate = Baud rate * number of bits per baud. Baud rate can be any value, but the only requirement is both the devices should be configured with the same baud rate for communication to happen. Common baud rates are 1200, 2400, 4800, 9600, 19200, 38400, 57600 and 115200 We will be using PL2303 (USB to Serial Converter) for this Serial Port Series. When you connect PL2303 on Linux OS, ttyUSB* device will be created. In our case, /dev/ttyUSB0 is created struct termios structure declared in <termios.h> holds all the settings for a specific port on the tty device. The input and output baud rates are stored in the termios structure. input flags control the input of...

ftrace provides "wakeup" tracer to find the scheduling latency. This tracer only records the latency for the highest priority to get scheduled after it woken up. Scheduling latency: Time from the wakeup event until the task actually gets the CPU and begins to run. You can see from the above screenshot, i wrote a simple program which calls "sleep" to voluntarily leave the CPU from the 'wakeup' process and to give it to another process for us to find out the scheduling latency. It took '30us' for scheduling the next highest priority task.

1. Starting up gdb : gdb <Executable name with debugging symbols> If you run only gdb, you can still specify file name using the 'file' command $gdb (gdb) file </path/to/executable> 2. Help of command: To get help of a particular command use "help" command (gdb) help [command] 3. Running the program:  (gdb) run 4. Setting breakpoints: Breakpoints are used to stop the program run in the middle, at a designated point. (gdb) break file1.c:10 The above command puts breakpoint at line 10 of file1.c (gdb) break <func_name> The above command puts breakpoint at the starting of this function 5. Continue the Program: After breakpoint, if you type 'run' it will start executing from beginning, instead of that to continue from the breakpoint, use "continue". Program will run until the next breakpoint or program termination (gdb) continue 6. Single Stepping: After breakpoint, if you want to single step to next line, instead o...

When interrupts are disabled, CPU cannot react to any external events: timer interrupts mouse interrupts irqsoff is a tracer which tracks the times in the kernel when interrupts are disabled. Whenever a new maximum latency is found, the tracer saves the trace leading up to that latency point. Each time a new maximum latency is reached, the old saved trace is discarded and the new trace is saved. By default this tracer is not enabled in my kernel, I had to re-compile my linux kernel with the following kernel configuration: "CONFIG_IRQSOFF_TRACER" You can see from the above screenshot, the maximum latency found was 644ms, started at _raw_spin_lock_irq.  Also , 'tracing_max_latency' file also provides you the maximum latency when the interrupts are disabled. Next time, trace file will be updated when it finds a latency greater than the value present in 'tracing_max_latency' file. To reset the maximum latency, write '0' in 'tracing_ma...

ftrace uses 'tracing_thresh' file and compare the function duration if  greater than the value present in the file, it logs tracing entries in the ring buffer. By default the value is zero, it means ftrace doesn't care this. The value written in 'tracing_thresh' is in microseconds. So, if you want to find out the kernel functions which are taking more than 10ms , you need to write echo '10000' > tracing_thresh Set back zero in tracing_thresh, if you want to trace all the functions. echo '0' > tracing_thresh

In the previous post we have seen how to find out the backtrace of a particular function. In this post, we will learn on what all functions are being called by a particular function. The file for this is 'set_graph_function' and tracer is 'function_graph' Let's find out what all calls are made by 'vfs_read'. You can see from the above screenshot, vfs_read calls many functions for a read operation

With ftrace, you can find out who is calling a particular kernel function. "function" tracer contains an option which creates backtrace for each function called by tracer. If you enable this option without any filter, chances are that your system will get into live lock. Note: You can use this feature only when the tracer is "function". Let's track who all calls "vfs_read". From the above steps, I did the following:  Set the tracer to "function"  Added "vfs_read" as a filter Enabled the function back trace option in "options" directory You can see the flow: system call -> ksys_read -> vfs_read. 

We know 'tracing_on' file is used to disable/enable writing the traces on the ring buffer. echo '0' > tracing_on //to disable tracing echo '1' > tracing_on // to enable tracing ftrace also provides functionality to start tracing on a particular function call execution or stop tracing on a particular function execution. Syntax: function:command[:count] The command will be executed on the start of the function. The value of command can be "traceon" to start the trace or "traceoff" to stop the trace, or it can be "stacktrace". The count is optional, specifies the number of times the command should be executed. If there is no count specified, the command will be executed each time the function is called. From the above screenshot, i told ftrace to stop tracing on ksys_read, which is called when user space calls read, and start tracing on ksys_write, which is called when user space calls write.

In our previous post, we discussed how to trace only specific kernel functions by echoing them into "set_ftrace_filter" file. To list what all functions the ftrace can trace, you can read "available_filter_functions" file. $ wc -l available_filter_functions 38750 available_filter_functions You can see from the above output there are more than 38000 functions which the ftrace is tracing if we don't add filters. The tracing files support wild cards,  the following expressions are valid: value* - Will select all functions that start with string "value" *value - Will select all functions that end with string "value" *value* - Will select all functions that start and end with string "value" As the star (*) is used by bash, you have to wrap the string in quotes when using wild cards. To trace all functions of a specific module, for example e1000 (Ethernet driver) you can pass the following to set_ftrace_filter. ...

What are anonymous structures/unions? Structures/unions with no names are called anonymous structures/unions. They are also called unnamed unions/structures. As there are no names, we cannot create variables of anonymous structures/unions, but we can use them in nested structures or unions. Real World Example of anonymous structures/unions: A status register of a microcontroller can have multiple bits. Sometimes we read the value of each bit, and sometimes the whole register. We can use anonymous structure/unions in this to easily refer to both. Code: Output: $ ./anonymous Status register value:0x83

'ldd' Linux command is used to print all shared library dependencies. Syntax: ldd <executable_name> Some versions of ldd will execute the application in order to get dependencies information. Safer alternative is to use: objdump -p <executable_name> | grep NEEDED Another way is to use readelf. readelf -d <executable_name> | grep NEEDED Note that libraries depends on other libraries, so you need to find their dependencies.

I wrote a script to find out what all kernel functions are being called for a particular linux command we execute. Script: Output: 

Building the Linux Kernel is a two-step process: Configuring the Kernel options Building the Kernel with those options. There are many methods available for configuring the kernel. make config: Text-based Configuration.  Options are prompted one after another. All options need to be answered Access to former options is not possible make menuconfig: Menu-driven user interface Allows to navigate forwards and backward directly between features Allows to load and save files with filenames different from ".config" Provides search feature It uses ncurses library for GUI. If the ncurses library is not installed, make menuconfig option fails. To install ncurses library on Ubuntu: sudo apt-get install libncurses5-dev make defconfig: Creates a ".config" file with default options from the ARCH supplied defconfig Configurations are generally stored in the directory: arch/$(ARCH)/configs $ ls arch/x86/configs/ i386_defconfi...

ftrace internally uses a ring buffer of fixed size. The size can be find out by reading "buffer_size_kb" file. If the buffer becomes full, it will overwrite the starting tracing entries. $ cat buffer_size_kb 1410 To read this ring buffer, we can cat either of the two files present in /sys/kernel/tracing folder: trace trace_pipe What is difference between trace and trace_pipe? Difference between the two files is that "trace" file is static. Each time you read the contents of "trace" file it will print the same tracing entries, whereas with trace_pipe, each time you read the file the data is consumed, and in the next read data will be different also if there is no data trace_pipe will block. Let's try to understand by running the commands. You can see, I did the following steps: Enabled the "function_graph" tracer Then I read both the "trace" and "trace_pipe" files, it displayed the sam...

ftrace has introduced a new form of printk - trace_printk. The problem with the current printk() is Using printk in interrupt context such as timer interrupts, scheduler, network can create a live lock Sometimes bug disappear when we add few printk's if something is time sensitive printk when writing to the serial console may take several milliseconds With trace_printk writing will be in the order of microseconds as it writes to a ring buffer instead of console can be used in any context (interrupt, scheduler, NMI Code) can be read via the 'trace' file Let's update our Keyboard character press driver code to use trace_printk instead of printk. Code: Output:

With ftrace, we can trace kernel calls only related to a specific process by echoing pid in the set_ftrace_pid file. Steps to trace kernel calls of a particular PID: 1. Disable tracing by echoing "nop" to "current_tracer" file $ echo "nop" > current_tracer $ cat current_tracer  nop 2. Echo the pid of the process which you want to trace in "set_ftrace_pid" file. 2588 is the pid of "gnome-terminal" $ echo "2588" > set_ftrace_pid $ cat set_ftrace_pid  2588 3. Enable either "function" or "function_tracer" by echoing into "current_tracer" file $ echo "function_graph" > current_tracer  4. You can now "cat trace" to see what all kernel functions are being executed by the gnome-terminal. You can also find out what all kernel function calls being executed by the running bash  $ echo "$$" > set_ftrace_pid The ...

Information provided by "function" tracer is a bit hard to follow. "function_graph" is another which tracks the entry of the function tracks the exit of the function Execution Time CPU on which it is running With "function graph" tracer, you can easily follow the flow of execution within the kernel To use function graph, $ echo function_graph > current_tracer and then cat the trace file. Start of the function is denoted with '{' and end of the function is denoted with '}'.  Functions that do not call any other functions, simply end with ';', also called as leaf functions Duration column reports the time spent in the corresponding function. These numbers are only for the leaf functions, and the '}' symbol. When the duration is greater than 10 microseconds, a '+' is shown in the DURATION column When the duration is greater than 100 microseconds, a '-' is shown in DURATION column ...

To enable function tracer, echo "function" to current_tracer file. $ echo "function" > current_tracer $ cat current_tracer To see the kernel trace, cat "trace" file You can see from the cat output: First line specifies the tracer installed, in our case "function" Next each line prints the following information: Process Name PID CPU that the trace executed on Timestamp in seconds with the decimal places to microseconds. This timestamp is time since boot Function being traced Parent that call the function

What is Tracing? From Wikipedia, tracing is specialized use of logging to record information about program's execution. This logged information is used by programmers for debugging. What is ftrace? ftrace stands for function tracer. ftrace allows you to trace kernel function calls (entry and exit of function calls). What can we do using ftrace? Debugging Linux Kernel Analyzing Latencies in Linux Kernel Learn and observe the flow of Linux Kernel Trace context switches Length of the time the interrupts are disabled What is the minimum kernel configuration required for ftrace? CONFIG_FTRACE --> "Tracers" CONFIG_FUNCTION_TRACER --> Kernel Function Tracer CONFIG_FUNCTION_GRAPH_TRACER --> Kernel Function Graph Tracer CONFIG_DYNAMIC_TRACEV --> Enable/Disable ftrace dynamically How to check whether ftrace is already enab led or not? Most of the Linux distributions already provide ftrace, to check verify "tracing" dir...

What is __init? __init is a macro defined in linux/init.h file #define __init      __section(.init.text) __cold  __latent_entropy __noinitretpoline What is the use of __init? This macro informs the compiler to put the text for this function in a special section named ".init.text". This way,  all the functions with __init are placed in a single ELF Section. This section is removed after all the init functions with __init are called. You can see the following line in "Freeing Unused Kernel memory" in dmesg output, which is nothing but freeing up all the pages of init.text section. This is performed by function free_initmem() present in arch/x86/mem/init.c, each and every architecture will have their own free_initmem() definition The linker script merges all "init.text" sections into a single section and assigns an initial address to it.  The linker script is present in "arch/x86/kernel/vmlinux.lds.S" . You can run ...

Yocto, while building images normally downloads the required files from Internet and places it in Internet folder. We cannot directly share this folder for other developers to avoid downloading files again, as this contains symlinks. To make downloads to store on a private repository, add the following in your local.conf file: INHERIT += "own‐mirrors BB_GENERATE_MIRROR_TARBALLS = "1" Run your "bitbake target" which will fetch the download files and build the target. Now, let's copy the downloaded files into another directory resolving symlinks with the following command: rsync ‐vaL <download_folder>  $/home/embedded/src_mirror/ Push this src_mirror on to a private git repository, for others to clone. Now, when some other developer has cloned this download repo on their machine, they can add the following configuration in their local.conf to use this downloads and avoid fetching any files from the internet. SOURCE_MIRROR_URL ?= ...

In our previous post of module stacking , we developed two modules: One module exporting a symbol Other module using that symbol We used insmod to load the first module which exported the symbol and then the second module. In this post we will see how we can use modprobe on the second module and the first module should get automatically loaded. Step 1: As modprobe reads only modules present in /lib/modules/$(uname -r), create symbolic link of your modules to /lib/modules/$(uname -r). Note, we need to pass full path of the module $ sudo ln -s /home/jamal/ldd/module_stacking/module1.ko /lib/modules/$(uname -r)/kernel/drivers/misc $ sudo ln -s /home/jamal/ldd/module_stacking/module2.ko /lib/modules/$(uname -r)/kernel/drivers/misc Step2: Update modules.dep and map files $ sudo depmod -a Step3: Run modprobe on the module which has dependencies $sudo modprobe module2 You can see when we did modprobe module2, module1 was also loaded automaticall...