In this post, I will walk you through the user interface and related parts of interesting project I developed: a Robot Keyboard PC Controller Interface. This interface uses an Arduino to emulate a virtual keyboard and a Kotlin client for communication. The primary aim is to provide a cross-platform solution that works on Windows, with potential compatibility for other operating systems.
Project Overview
The full project consists of two main components:
- Arduino Sketch: Emulates a virtual keyboard.
- Kotlin Client: Connects to the Arduino via serial communication, sending data from the PC. It supports both command-line and GUI interfaces for user interaction. The simple GUI interface is shown below.
Key Features
- Serial Communication: Connects a Windows PC to an Arduino Due using serial communication.
- File Watching: Monitors a file for commands, making it compatible with the StreamDeck interface and AutoHotkey scripts for enhanced flexibility.
- Local Typing: Uses Java’s AWT Robot to simulate typing on the local machine, useful for testing purposes.
Reasons for choices
I want to be able to test it from end to end, and also control the Arduino, or the pc to which it is pretending to be a keyboard if something goes wrong, so It has a simple user interface, both command line and GUI.
Also to help test out the virtual keyboard I decided it would be useful if it could type on the local machine too, (note that this local machine keyboard is a simplified version of the Arduino that does not support all functionality, it would be possible to extend it but I really have no need to. The local keyboard typing uses Java’s AWT Robot, and for this reason I chose Swing for the GUI. The AWT Robot can type into other applications, unlike some more modern choices. AWT robot is well supported on Microsoft Windows but may not be well supported elsewhere, particularly on Linux.
Configuration
The project is configured using a application.yaml file:
application.yaml
spring:
application:
name: robokey
server:
port: 8080 # Port for the REST controller
error:
whitelabel:
enabled: false
app:
consoleCommands: true
gui: true
darkMode: false
mode: PHYSICAL
comPort: COM4
logFilePath: logs/dummy.log
keyboard:
allJitter: true
output: true
uiExtraDelay: true
initialDelay: 3000
press:
jitter: false
jitterMax: 100
time: 100
delay:
jitter: false
jitterMax: 300
time: 300
pipe:
useOsPipe: true
unixPath: /tmp/robo_keyboard_named_pipe
fileWatcher:
enabled: true
winfilePath: "C:\\data\\robo_keyboard_commands.txt"
posixFilePath: /tmp/robo_keyboard_commands.txt
createFile: true
charset: UTF-8
fallBackCharset: UTF-8
Code Highlights
Modern Look and Feel with FlatLaf: I used the FlatLaf library to provide a modern look to the Swing GUI.
implementation("com.formdev:flatlaf:3.4.1")
Serial Communication with PureJavaComm: For serial communication, I used the PureJavaComm library.
implementation("com.github.purejavacomm:purejavacomm:1.0.1.RELEASE")
Opening Serial Port: Here’s how I open and manage the serial port communication, I use coroutines to allow me to read and write at the same time:
val portId = CommPortIdentifier.getPortIdentifier(portName)
if (portId.isCurrentlyOwned) {
if (commPort != null) {
println("Using existing serial connection.")
} else {
println("Error: Port is currently in use")
return
}
} else {
commPort = portId.open("SerialPortApp", 2000) as SerialPort
commPort?.setSerialPortParams(
9600, // Baud rate
SerialPort.DATABITS_8,
SerialPort.STOPBITS_1,
SerialPort.PARITY_NONE
)
out = commPort?.outputStream
`in` = commPort?.inputStream
}
scope.launch {
readFromSerialPort()
}
Local Typing with AWT Robot: To simulate typing on the local machine:
val robot = Robot()
text.forEach { char ->
when (char) {
'\n' -> {
if (appConfig.keyboard.output) {
robot.keyPress(KeyEvent.VK_ENTER)
Thread.sleep(50)
robot.keyRelease(KeyEvent.VK_ENTER)
} else {
print("KeyEvent.VK_ENTER, ")
}
}
// ... rest of code
}
// ... rest of code
}
Spring Boot Application: The application is a Spring Boot application with both GUI and command-line interfaces. I used a TaskPool to allow me to execute multiple threads as some were unsuitable to be coroutines. I try to handle clean shutdown be sending each task a cancel message in my ThreadPoolManager , after which I can try to shutdown the thread pool cleanly and if that fails force it. This code has to set headless to false very early, it also does it more than once for safety.
@SpringBootApplication
class RobokeyApplication(
val appConfig: AppConfig,
private val swingMainWindow: SwingMainWindow,
private val consoleCommandHandler: ConsoleCommandHandler,
private val namedPipeCommandHandler: NamedPipeCommandHandler,
private val fileMonitorCommandHandler: FileMonitorCommandHandler,
private val taskPoolManager: TaskPoolManager
) {
@Bean
fun runOnStartup() = ApplicationRunner {
if (!GraphicsEnvironment.isHeadless()) {
swingMainWindow.createAndShowGUI()
}
if (appConfig.consoleCommands) {
taskPoolManager.submitTask(TaskName.CONSOLE_COMMANDS) { consoleCommandHandler.start() }
}
if (appConfig.pipe.useOsPipe && !System.getProperty("os.name").lowercase(Locale.getDefault()).contains("win")) {
taskPoolManager.submitTask(TaskName.PIPES) { namedPipeCommandHandler.start() }
}
if (appConfig.fileWatcher.enabled) {
taskPoolManager.submitTask(TaskName.FILE_MONITOR) { fileMonitorCommandHandler.start() }
}
}
// ... rest of code
@PreDestroy
fun shutdown() {
taskPoolManager.shutdownAndExit()
}
}
fun main(args: Array<String>) {
val log = LoggerFactory.getLogger(RobokeyApplication::class.java)
if (args.contains("--help") || args.contains("-h") || args.contains("/h")) {
ConsoleCommandHandler.help()
// hasn't started up the taskPool yet so can just exit
SystemExitHandler().exitProcess(0)
}
log.info("Starting app ...")
System.setProperty("java.awt.headless", "false")
runApplication<RobokeyApplication>(*args)
}
REST Controller: Provides a simple REST API for sending commands. Here is an except showing some functionality
@RestController
class KeySendController(val keySendService: KeySendService) {
@GetMapping("/ping")
fun ping(): Mono<String> = Mono.just("OK")
@GetMapping("/command")
fun sendCommand(@RequestParam("text") text: String): Mono<String> {
return Mono.defer {
Mono.fromCallable {
URLDecoder.decode(text, StandardCharsets.UTF_8.toString())
}.subscribeOn(Schedulers.boundedElastic())
}.flatMap { decodedText ->
Mono.fromCallable {
keySendService.sendCommands(listOf(decodedText))
"Command sent: $decodedText"
}.subscribeOn(Schedulers.boundedElastic())
}
}
// ... rest of code}
}
Kotlin Coroutines vs Threads
I used a TaskPool to allow me to execute and control multiple threads. I did this as some were unsuitable to be coroutines. Kotlin Coroutines are a tool for writing asynchronous and concurrent code in Kotlin. They allow for non-blocking code that is simpler and more efficient than traditional threading. I should avoid using coroutines and thread at the same time as there can be problems doing this. I do plan to refactor this before publishing full source, and will either need to switch everything to threadpools, or design my application so that coroutines will work, this might be a good route as splitting things up may allow me to send higher priority messages to interupt work in progress.
The Primary Use Cases of Coroutines are:
Asynchronous Programming: Writing asynchronous code in a sequential manner.
Concurrency: Performing multiple tasks concurrently without explicit thread management.
Background Processing: Running tasks like network requests or file I/O in the background.
UI Updates: In Android, performing background tasks while updating the UI on the main thread.
Benefits of Using Coroutines for Managing Concurrency:
Simplicity: Write asynchronous code without callbacks, making it more readable and maintainable.
Efficiency: Lightweight and efficient, handling many more concurrent tasks than traditional threads.
Scalability: Scale better than threads, handling thousands of tasks with lower overhead.
Structured Concurrency: Manage the lifecycle of concurrent tasks, preventing memory leaks.
Error Handling: Built-in mechanisms for handling exceptions in asynchronous code.
Examples of Tasks Unsuited for Coroutines:
Blocking I/O Operations: Tasks that involve heavy blocking I/O operations like reading large files or blocking network calls.
Long-running CPU-bound Tasks: Intensive CPU-bound tasks that run for a long time may benefit more from a dedicated thread pool.
Real-time Systems: Systems requiring strict timing constraints where coroutine scheduling might introduce unacceptable latency.
Tasks with High Precision Timing Requirements: Tasks requiring high precision timing, such as certain simulations or hardware interactions.
Tasks Requiring Synchronous Execution: Tasks that must be executed in a strict sequence without interruption, where coroutine overhead might not be justified.
Conclusion
This project is an example of integrating Kotlin, Arduino, and Java technologies to create a versatile robot keyboard PC controller interface. While I haven’t published the full source code yet, I plan to do so after further refinement and clean up. Stay tuned for more updates and detailed explanations of my other projects!